Daniel Quinn

I’ve had a really hard time figuring out how to get cloud native pg working 'cause I couldn’t get longhorn working for disk space.

So instead I went with a separate Raspberry Pi that isn’t part of the cluster to host a single Postgres instance.

It’s inelegant, but has worked for years. Still, I’d rather host a separate pg instance for each project… I just have to figure the above out first.

#Solarpunk!

I’ve used FluxCD in the past and have looked into ArgoCD, but honestly, I’ve not seen any big benefit from either to be honest. I use k8s both at home and at work, and in both cases, we do “imperative” deploys: you run helm install ... either directly or via the CI and stuff is deployed.

So for example at my last job, our GitLab CI just had a section triggered exclusively for merges into master that ran helm install ... for all three environments. We had three values.yaml files, one for each environment, and when we wanted to deploy a new version, the process was:

1. Create a tag for our release version (ie. 1.2.3) and push it to the repo. This would trigger a build and push the resulting image into the container registry.
2. Push an update to the repo with the new tag set in the appropriate Helm values file. If we wanted to deploy 1.2.3 to development but not yet to staging or production, then the tag: value in each of the environment files would look like this:

• k8s/chart/environments/development.yaml: tag: 1.2.3
• k8s/chart/environments/staging.yaml: tag: 1.2.2
• k8s/chart/environments/production.yaml: tag: 1.2.2

Once that change is pushed, the CI will automatically apply it with helm install ... and make sure that all three environments are what they’re supposed to be.

As for dependent services, that should all be in your Helm chart so they’re stood up and torn down together. The specific case you mention about “Service A” being dependent on “Service B” but stood up before “Service B” is ready is a classic problem, but easily solved:

The dependent service (“A” in this case) should have an entrypoint that checks for everything else before starting. Here’s what I’m using right now in a project:

#!/bin/sh

while ! nc -z "${POSTGRES_HOST}" 5432; do
  echo "Waiting for postgres..."
  sleep 0.1
done
echo "PostgreSQL started"

touch /tmp/ready

exec "$@"

I’ve even got some code that checks that all the Django migrations have run first for the same situation. The Kubernetes philosophy is that any container should be able to die at any time and be eventually be brought back up and that every container needs to be prepared for this. Typically this means that your containers should operate on the basis of “if I can’t work, die, and hope the problem is solved by the time Kubernetes redeploys me”.

Kubernetes. For a homelab, the stripped-down k3s is fantastic and surprisingly easy to get going.

Once you’ve got Kubernetes set up, you can lean on all the many tools already out there for things like deploying complex projects (Helm) and monitoring (Prometheus/Grafana). OpenLens is a nice piece of software you can use to monitor and control your cluster too, as is k9s.

Honestly, I’d buy 6 external 20tb drives and make 2 copies of your data on it (3 drives each) and then leave them somewhere-safe-but-not-at-home. If you have friends or family able to store them, that’d do, but also a safety deposit box is good.

If you want to make frequent updates to your backups, you could patch them into a Raspberry Pi and put it on Tailscale, then just rsync changes every regularly. Of course means that wherever youre storing the backup needs room for such a setup.

I often wonder why there isn’t a sort of collective backup sharing thing going on amongst self hosters. A sort of “I’ll host your backups if you host mine” sort of thing. Better than paying a cloud provider at any rate.