I wait for something to break. Then I yell “God fucking dammit, I don’t have time for this right now!” and spend an hour triaging things before I just try docker down, pull, up and everything works.

Poorly.

I just use ssh and manually type commands. Keep doing it and you will get good and it will become second nature.

No need to burn tokens on basic tasks.

Master your tools. Learn awk, sed, just, etc.

Some shell customisation can help. For example I’m fond of zsh-auto-suggestions and skim. Makes me quicker.

I use ssh with tmux

I created git repos on my main workstation for each homelab server/service I maintain that keeps:

• documentation
• notes
• lessons learned
• scripts, configs
• runbooks
• backup details
• security audit details
• log items that need attention
• infrastructure

I just point a local LLM (offline model that runs on my workststion) into those repos and ask it to perform certain things on those servers. It can do things like update packages, install packages, make config changes, set/check permissions, read logs (and fix errors in real time), and check the health of the overall system.

I have it run pre backups before making changes, then post backups once its done.

Once changes are in place and everything is running okay, I ask it to update documentation in the repo and tag the release.

I use opencode that connects to a llama.cpp service. opencode lets me gate the AI so that any elevated commands that it needs to run (e.g. sudo or ssh), I have to approve it. It cant just go around making changes without permission.

Ansible.

Maybe a bit more on the overkill side, but I use Talos Linux (declarative K8s distribution), then Terraform for initial setup and FluxCD for everything else (including VMs via Kubevirt). It’s a hell of an initial learning curve, but afterwards I just do my talosctl upgrade and upgrade-k8s from time to time, and don’t have to worry about anything else. Upside is, Kubernetes provides a unified, extensible API for everything, for example:

• reverse proxy via Ingress or Gateway-API
• firewall via NetworkPolicy
• even databases like Postgres through an Kubernetes Operator like CNPG, with a similar simplicity as with the big cloud providers (just a single yaml file with the specs like storage capacity, CPU and memory limits, backup target and schedule etc.)
• it is very scriptable (everything is managed via the API)
• automated image updates via either FluxCD itself or just dependabot/RenovateBot creating PRs to your GitOps repo

Downsides:

• you have to figure out persistent storage (CSI), which is slightly more complicated than just a single filesystem and manually specifying volume mounts like with docker, but if done right, you have a simple interface with powerful capabilities via the Kubernetes API
  • there are simple CSI implementations that just expose node local disks via LVM or ZFS, so if you just have a single node, or don’t care about replication, it’s fairly easy to get started
• initial learning curve is quite high, especially if you have no prior experience with container orchestration in general
• definitely overkill if you just want something simple that “just works”

nixos, mostly. It has a very steep learning curve but if it breaks I’m just “whatever, rollback now, fix properly later”.

I’ve been using a program someone on Lemmy has been writing: SSHPilot. It helps keep my server list organized and can easily ssh to the machines or open up a file browser for easy drag and drop of files if I have some quick files to move around. It copies ssh keys easily as well.

Nothing wrong with good old ssh. My VMs are different enough that building and maintaining ansible playbooks or something similar would be a bigger task than actually doing it in the traditional way. Maybe I could benefit by building simple playbooks to populate user accounts, ssh keys, smtp relay settings and other common things, but I don’t really set up new servers that often that it would justify spending time to set up tools for that.

Ansible and terraform can bath do with you want though for a single server I’d consider them both to be a somewhat overkill. But then again homelabbing often is about overkill.

Depends what you’re trying to do, really

If you’re talking about going in and restarting containers and checking logs, there are tools like Portainer that give a nice UI you can quickly use from your phone

If you’re talking about repetitive tasks, sometimes there’s a tool to automate it already (e.g. updates via watchtower, unattended-upgrades), and if not, you can write a script and possibly run it via a cronjob if that makes sense

If you’re talking about doing new stuff, you either stick with the SSH & shell commands, or you look into IaC tools like ansible as another reply mentioned. Another option is something like Nix where the setup is defined in config, so you make your changes there

I have a kubernetes cluster inside of Proxmox vm’s, generally I write a deployment, commit it to a git repo and then use argocd to deploy it in the cluster.

This is not something I would advise to anyone but the clinically insane, but for me there is a certain zen in having everything in git like that. It’s for me the only way I can manage the 60ish applications I run without it having a permanent spot in my brain.

My target is terraform to provide VMs on my miniPC and ansible to configure them. For my nas, probably a basic distro then some ansible stuff to setup stomate.

I spent a while getting Ansible to be able to setup and maintain my server(s). And ended up not using it as much as I should, mostly because the computer I used to run Ansible from became a “Steam Machine” so I’m rarely sitting on it with a keyboard now and I don’t want to have personal info and keys on my work computer.

But it was a good learning, and useful while I used it. Now I just use ssh and compose files directly.