Hello self-hosters! :) I’d like to tap into your experience
tl;dr: Is it really a bad idea to use MariaDB with data over NFS?
My setup
- RAID NFS NAS on RPi4
- “CPU machine”, just an old laptop for now
The idea is that NAS stores the data and RAID mitigates single point of failure. Laptop is only for running the stuff, all docker-compose are under git, uploaded elswhere. Laptop fails - I buy a new refurbished one and put everything back together from data on NAS and git
Problem
But Nextcloud uses DB. ATM it sits on the laptop. And I’m thinking how to somehow move that data to the NAS.
At the beginning NAS was just for NC files, connected as NC “external storage” over SFTP. But there were issues with this setup and NFS4 turned out to not require Kerberos to work.
Since NFS docker volumes have been working quite fine, I thought about just moving the data to the NAS. But then I read this:
Generally, we recommend not to use NFS (Network File System) with MariaDB, for these reasons:
MariaDB data and log files on NFS volumes can become locked and unavailable for use. Locking issues may occur in cases where multiple instances of MariaDB access the same data directory, or when MariaDB is shut down improperly, for instance, due to a power outage. In particular, sharing a data directory among MariaDB instances is not recommended.
Data inconsistencies due to messages received out of order or lost network traffic. To avoid this issue, use TCP with hard and intr mount options.
And I feel discouraged
- I could move whole DB (I mean including the process) to the NAS, but then operating the DB becomes NAS’ burden and breaks the responsibility separation
I’m not a fan of this idea - I could treat NAS as first stage of backup for that DB data and just send the data to NAS once a day
Also a bit meh. Getting back up from server failure will require copying the data - Maybe it’s not that bad to use MariaDB with NFS?
If there is a problem with server-NAS connection, I actually don’t want Nextcloud to work at all. It should fail, so I know I need to fix something
So, what are your experiences? Is running the DB on one machine, using the data from another machine over NFS a shot in the foot?
You must log in or register to comment.
It’s okay for small scale, dev, etc…on a fast and flat/local network.
But databases really don’t like NFS, SMB, or most file sharing protocols. There’s too much transaction overhead.
The effect gets exponentially worse as you add in latency. Database may as well be on the moon if you’ve got more than 10ms between it and the app (hyperbole but not really).
Use fast local storage, ideally. Or block if you have it. Async replicas if you are chasing 9s.
When databases fail, they fail spectacularly and cause ripples. Remember that you’ll be the one supporting it when they do.
Eh, I’d take their recommendation with a grain of salt. They’re likely targeting worst case. Like a university with thousands of users. If you say put proxmox on the laptop, connect your nas as storage, then run the db vm/container with it’s storage as the nas, I doubt you’d see any issues. You might notice a bit more latency and slowness, but I doubt any data loss. This setup is how I run all my VMs for years, some of the VMs have databases. Zero issues.
Without getting into things like a SAN it’s generally not a good plan to put anything that needs realtime access remote to the device processing it. DBs are constantly making updates and introducing the odd bit of latency or a disconnect can make for lots of issues. I also have a worker/storage split setup and for a while tried just running the VMs as a whole using remote storage to host the vdisk, it wasn’t pretty even with a separate network dedicated to it.
For NC particularly, if you use their official AIO docker setup there is a built in borg backup you can store remotely and it makes restoration pretty painless.
I would echo this, dont put your database on NFS. it will be slow and if the link goes down, it could destroy the database.
Agree. I’m still not doing proper backups on my homelab but this is what i’m gonna focus on next. OP has a good NAS setup and IMHO the focus should be on backing up the DB for later restoration (if there’s the need because it might never happen) and keep the DB on the most powerful CPU and quickest disk.
keep the DB on the most powerful CPU and quickest disk
Thing is, I expect that NAS to be faster than the SSD on the laptop that runs a bunch of stuff. Even NC log file will take up some IO
But I get the point you are all making here, I’m changing course :)
it’s generally not a good plan to put anything that needs realtime access remote to the device processing it
So in your opinion putting whole
nextcloud_dataas NFS mount is also a bad idea? Keep NFS as external storage and/or only user data?AIO docker
It was too blackbox to me. I had some issue and found it hard to debug. fpm+nginx is more my speed
This BorgBackup app is available only in AIO?Borg is its own thing, just handy to have it built in and preconfigured to work. If you run VMs on your working host instead of running NC directly on the machine you could just back those up for pretty much the same effect, though it wouldn’t usually be a realtime backup.
The next cloud data is fine being remote, when I say realtime I guess it could be better said as time sensitive. A database needs to update something and read it back in the right sequence for it to work properly. Just retrieving a file doesn’t have those issues.
If you happened to use it in a way like shared collaboration of a document with realtime updates it might be an issue, but I expect in that case the revisions would hit the DB first and then be put to the file as a secondary operation. Never tried that myself though.
I have never attempted this but to me it sounds like it might lead to weird issues if nfs is slow for some reason and it will definitely make your nextcloud instance slower. You can easily just try it out though. Copy the docker compose project, start another mariadb instance with its data directory mounted from the nas. That way you can figure it out yourself.
Personally I would just back up the database to the nas regularly. Yes if your laptop fails this requires copying data. You already need to install an os on the new laptop and pull the git repo so having one more step of copying the data should be easily doable in my opinion. Especially since you hopefully don’t plan on having your laptop break all the time.
You can easily just try it out though
Yeah, but if that setup leads to problem in a few months, I won’t catch that. That’s why I decided to ask and judging by the other answer, I dodged a bullet here
Now I’m thinking of setting up something like this. Maybe even in another container? One volume with data bound, one with NFS and a bunch of scripts in entrypoint to rsync(?) between the two on inotifywait. There is lsyncd but it’s been unmaintained for two years now
Not nextcloud, but I have several containers set up to use ‘third party’ database - ie, not running inside the container or compose group, sometimes on a second machine. It definitely complicates the setup. I haven’t noticed any performance issues, but my services are low load.
I would think that the db running on the Pi with storage on the Pi could be faster than the db running on the laptop with storage on the Pi. The Pi db would presumably have higher bandwidth to the Pi storage and only have to transfer the selected data over network, where the laptop db has to do all the db sorting, filter, and processing over the network, although I suppose the db would try to keep all the data in RAM. I know there’s limitations to Pi’s internal bus bandwidth, Maybe you can compare the hdparm performance of the NAS RAID vs the NFS mount.
db running on the Pi with storage on the Pi could be faster than the db running on the laptop with storage on the Pi
I think so too, but I’m mindful of the 1GB RAM per TB rule. I’m afraid to choke the NAS and on Pi I can’t put in more RAM
