I am fairly new in the selfhosting business. I host a few services like Jellyfin or AdGuard, but most of my data just sits on plain SMB shares on my NAS.
I am interested in setting up things like Paperless-ngx, Nextcloud maybe Immich as well. One thing I am struggling with is how to organize all this so I don’t have the data scattered around multiple places or in the worst-case even duplicated. I like the simplicity of network shares and the fact that I am not relying on a third-party application to keep being maintained. I am currently not sure I am willing to give that up. However, I am also intrigued by the features these services provide. I know Nextcloud has the option to mount external storage, but I don’t know which downsides come with that. It gets more complicated with Paperless. As far as I know you typically have a consume directory where you throw in your data and Paperless stores it using its own system in the media directory. This means I either throw in everything or I suddenly have two locations where my documents are stored. If I would mount that media directory in things like Nextcloud I probably wouldn’t be able to find anything because of the different structure and naming scheme.
The idea that I had in mind is that I have a single directory structure of my data that can be used on its own and all these tools are just different frontends and provide different views and information of the same data. Maybe this approach is just something from the past and I should move on.
I should add that I am not planning to expose any of these services to the public. All of this is only accessed from inside my house or using a Wireguard VPN.
- How do you guys handle all this?
- How do you avoid data duplication?
- How do you avoid multiple potential file locations? Is a document in my Nextcloud, in my Paperless or just on the network share?
- Do you prepare in some ways in case an application stops being maintained?
- How has this reliance on multiple services impacted other things e.g. your backups?
@5ymm3trY@discuss.tchncs.de Excellent questions. I’m fairly new at this but I try to be organized. This may be a bit ‘meta’ for your taste but here it goes.
I use: Portainer to manage containers (FOSS, self-host of course!)
Stacks: Group by service importance or structure/db type I research each service thoroughly with AI for stability, activity and how it integrates into my stack and homelab ‘philosophy’ (see below)
Obsidian: to keep detailed notes of the entire stack (with canvas diagrams to define worfklows and keep stacks organized). I also have a LifeMap (how my homelab supports my life) section which AI vets my new services against.
Claude: as my advisor fore everything, pointed at my Obsidian vault. Makes for a much faster roundtrip, as I edit things in Obsidian (pick your .md editor) and Claude just advises/rewrites back to obsdian on the fly.
Main Setup:
- Philosophy: Low wattage 24/7 prosumer mini-pc runs homelab Ubuntu
- Services: should be light, simple and integrate well with LifeMap
- LifeMap: All areas of my life monitored / integrated (Health, Productivity, Ideation/emergence, Content, Moods/Habits, etc… <put in yours>
Vault Folders:
- Match Stacks for ease of management/guidance from Claude
Stacks: dawarich - mapping w/immich photo integration (on its own)
ghost - blogging platform, obsidian drives content high priorit backups
immich - photos high priority for backups
media - other less critical media tools i can bundle together lower priority backups since most of this stuff can be refound audiobookshelf calibre-web-nextgen jellyfin navidrome - higher priority backup of music yamtrack
tools - all the little workers doing interesting things into one stack that I can reload often without impacting more important services excalidraw immich-kiosk immich-public-proxy immich-reversegeo immich_power_tools linkding maloja miniflux miniflux-db netdata nextflux ntfy uptime-kuma rclone watchyourlan ytzero
wanderer - on its own due to complexity and more active usage by a hiking community, back end …py scripts integrating with immich, dawarich, gpslogger GPX uploads
File Structure: (all on SSDs)
Audio Audiobook backups (just a staging area for dumping stuff, it all goes to B2 via rclone) Book Course Film Image Immich Music Obsidian-RSS Podcast Show Video
Jellyfin and Immich point to the same folders I use Immich folder tools (can elaborate) to move photos / videos to match this structure, so they can both ‘share’ the same items. I find immich a wonerful front-end to crawl thru endless family videos before I process them in a video editor
Having a unified location for all your media files is really great when you start integrating your whole world holistically. I use whisper-ai to transcribe podcasts (audiobookshelf), youtube vides (YTZero), etc. and I always know where everything is.
When I add a new service (say paperless-ngx or stirling-pdf, my next project), I first ask Claude to review my stack documentation, my homelab specs (and security, network, cloudflare external, etc. setup) and then help me decide what service to pick of the basket of FOSS document management tools. I also use sumguy.com articles to review pros and cons of various services, quite helpful.
Then I get Claude to give an implementation doc with .yml, envars and all the tweaks documented in Obsidian. Then I test, and when stable, finalize documentation and make sure Obsidian is updated, and Claude has access to that new ‘folder’ or stack information for the next round. That folder is like a MOC in Obsidian, a folder in Obsidian, a project in Claude and usually a folder on the SSD as well. I use the .canvas mindmaps to help me track it all.
Then, once in a while, I tell claude to build me a workflow document with all the .py scripts, cron jobs, etc. into a document. For example Health Workflow Pull data from fitbit, drop int Fitbaus, push to postgres, analyze with Grafana or similar. Then I have it convert that into a canvas document, which makes a mindmap of the whole thing I can’t tell you how helpful it is to push mindmap boxes around and update the workflow. Of course, n8n is next, I’m just not quite there yet. But n8n will be a breeze with all my documentation.
Good luck
I suggest to skip the nextcloud. It’s a mess. Use immich for pictures and paperless for documents.
If hosting some other files that don’t fit those 2 categories, things like filebrowser or copyparty are great. Nextcloud is really quite the challenge tbh and imo not worth the hassle it is for a typical selfhoster… It’s hard to get the most out of it. It pays of for medium organisations maybe but not for just you and a few other people…
To be honest, I don’t really have a use-case for it, because I don’t really need any of the sharing and collaboration features which are kind of the main point imo. The only compelling reason is that I read somewhere that it is a good option to backup mobile devices. I know that Nextcloud is a real beast of a software and there are probably some more lightweight options out there for that purpose.
I never heard about any of those two, but it looks like file browser is going to be archived in September if that is the one you mean.
Yes filebrowser is ending it’s a shame, very simple tool. Copyparty works fine too but too many features I don’t need.
I don’t have a better suggestion for mobile backup. I only have pictures (immich) backed up automatically. Everything else on my phone would be lost or it already sits somewhere else from the beginning (like music)
So your pictures are automatically backed up using the Immich mobile app? That’s probably good enough anyway. If you want to, you can always setup something like Syncthing to get other files as well.
I can imagine. I just skimmed through their Github page and saw the claim:
inverse unix philosophy – do all the things, and do an okay job
These swiss army knives of software can be challenging. But I guess that’s also true for Nextcloud, probably even more so.
Yes, immich sends all my pictures to the homeserver AND I can immediately see all my partners pictures pop up on my phone. Immich is very cool for that.
I too dislike that immich “scrambles” my pictures. I had it quite well organised over multiple decades (folders per year & location), but how well immich works seems worth losing this idea of organisational control over the collection. The immich duplicate detection for example is superb (it isn’t on nextcloud). It for example detects very easily that a 400kb sent or received picture is actually the same as a 4Mb much higher resolution original picture with a completely different filename and different metadata.
I still have my pictures collection organised old school from before immich on various back-up media and the hardest part to me now is deciding to keep that or convert it all to the “immich style” directory… Since it’s now very difficult to add new pictures to it without loads and loads of duplicates.
I did find the setting in immich which will somewhat still organise it in folders per month now, but it’s not the same, since a month often isn’t a logical grouping directory, for example if you go on holiday last week of December+ first week of January: pictures will still be spread across various folders.
My main downside on immich currently is that we can’t easily “share” metadata over a few different users (locations added later, recognised faces etc) although we share many faces/locations in pictures… It’s a hard choice wether I still want to make the few users all use the same account after all because of that (which will have other downsides like no personal private folder)
I didn’t know it has that feature, but that’s quite nice.
Yeah, i think it’s the same for Paperless and Immich. It is just a different approach. If you have tags, facial recognizion/OCR, some form of album/collections and other searchable metadata it doesn’t really matter how it’s organized in the filesystem because you can search it easily.
It’s this metadata that opens up the possibility of a separate frontend and backend. Whereas in the olden days it was backend all the way with meticulously naming files and folders.
It is probably better to let the tools handle it themselfs instead of forcing your old habits onto the tools.
Thanks for the whole writeup about your Immich experience. I am definitely coming back to it, when I test it out.
I run everything in docker. My docker volumes are all subdirectories under /media, which is the root of my NAS. If an app only needs something specific, docker only mounts, eg /media/movies. If the app needs many subdirectories (like backups), it gets the full /media mount.
All the docker data/configs are under /docker/volumes (eg /docker/volumes/jellyfin), and every container is configured with compose in /docker/docker-compose.yml.
The biggest hurdle for me was creating a sane docker-compose file that defines where everything lives. Once that is standardized, adding and maintaining your apps becomes easy.
If you’re very new to this, even a dumb LLM like ChatGPT can be very helpful to explain what’s inconsistent, redundant, or incorrect with your config. If you are lost I would strongly suggest this as a way to get oriented. But DO NOT copy and paste anything into the LLM containing passwords, tokens, keys, etc. if you accident do, change those keys before finishing up your project.
With docker there’s no duplication- you have one copy of everything you need and you just point each container at the same data. This makes it trivial to keep file consistent across your apps and doesn’t waste space.
I already use Docker and docker-compose for all the services I host. They are also all configured to use bind mounts instead of volume mounts, so that I am in control where everything is located. I mainly use the official documentation although I sometimes like to check https://github.com/Haxxnet/Compose-Examples/tree/main. I didn’t came across any major problems with that workflow so far, but that is also besides the point of my original question.
Jellyfin is a good example of the workflow I had in mind. I have a media share on my NAS and just point Jellyfin to that share in the docker-compose. But that is not how things like Paperless work, at least not from my (still limited) understanding of it. I think there are other workflows, but the main one of Paperless is a consume directory and a data/media directory. Everything in the consume directory gets ingested by Paperless and is moved with (I think) a different hierarchical structure and filename to the media directory. So if I point the consume directory to my current documents share on the NAS everything is moved away. I am sure there are ways to specify how Paperless organizes the media directory, but it most certainly will not be exactly the same as before. So making the data/media directory function as my new NAS “share” will also result in a change. In addition to that, my documents share on the NAS also contains data that I probably don’t need to track with Paperless e.g. some spreadsheets. I am not sure how Paperless handles documents in the media directory that were not originally processed using the consume folder or if that is even a good idea in the first place.
- Just like czl I do have seperate apps for each type of data I use. For example Ente for photos and Syncthing for keeping the documents folder of my devices synced up.
- It’s not something I really think about because my data sources are functionally disjoint. (e.g. I don’t have photos from my phone in my documents folder). The only exception is Ente, where I do keep a plaintext copy of the “shared with family” account to display some family photos on our digital photoframe. This also “answers” your next question.
- I don’t have anything set up for when an application stops support. I do however do look beforehand whether it’s possible to extract my data in a way I can migrate to another platform if need be. For example, some services like Ente offer an easy plaintext way to export/backup your data in directories mirroring your albums. That way I could easily head to Immich
- It hasn’t really. I usually do backups by keeping two copies of all my volumes. So running more services doesn’t really impact much of anything.
A sidenote on Nextcloud: it does use the same folder structure in its data volume as it presents to the user. So it’s fairly easy to manage, but you need to update the folder you made the change within the container for it to show up. When we used to keep photo’s synced up with Nextcloud, I could directly access that folder for the photoframe. Since Ente is e2ee I needed to duplicate data in that case
Yeah, that keeping things separate approach seems to be a good idea. It is just that idea that I can interact with my data from within various different tools, has some kind of appeal to me.
