I’m curious as to which tools and technologies you all are using to keep track of all those services you are deploying, whether it be resource tracking, network traffic, logs, traces, or uptime.
As a bonus question, how have you organized your network or your services to reduce the overhead of implementing observability?
If you’re wanting to go hard in the paint here, Grafana has a great OSS stack with Grafana, Loki, Mimir, Tempo, and Alloy.
Can confirm it’s a great stack! We run it at my work and its quite nice to work with. I’ve been thinking of replacing Prometheus in my homelab with Alloy
It’s curios if you used Prometheus as well
It looks really compelling. Really flexible with deep options. Seems like I could spend 10 years building the dashboard alone!
Observawhat? My services just exist, if they fail and nobody complains… Well, they get removed. If the fail and somebody complains… The get restarted…
Pretty efficient but simple approach.
I got tired of only noticing something was broken when I needed to use it instead of when I was working on things. Now if I make a change I notice issues right away and can fix them.
Things usually break on updates unless your disk fills up or worse some hardware fails…
So it’s pretty easy to fix: never upgrade automatically.
I always upgrade manually and check immediately what I have upgraded. Works fine so far.
And I was only partially joking… Yeah if a service stays broken for a long time, just uninstall it, it was useless.
Uptime Kuma if I want something to alert me (still trying to decide between ntfy & gotify)
Home Assistant is also checking some stuff
Watchdog to reboot my Raspberry Pi Zeros when they fall off the wifi
Smokeping for a general, long term feel of the network, which might answer your bonus question?
Did you ever decide on Ntfy vs Gotify?
Not yet 🙂
Both services work well, but the (Android) app is also a consideration… Gotify appears to keep losing the connection, but as I have different alerts going via different routes I need to sit down and spend some quality time with these, which I don’t have at the moment.
I asked since I recently started using Ntfy and was just curious if you found something in Gotify that’s relevant to me
I have different alerts going via different routes I need to sit down and spend some quality time with these, which I don’t have at the moment.
If you mean you send notifications to different services/platforms, you should look into Apprise, which is meant to do exactly this for you but easier supposedly
I asked this of the #homelab fediverse a few weeks back, as I was starting to deploy a family of apps to my “homelab” which is really a raspi3b with a repurposed spinny-platty laptop drive connected to some cloudflare tunnels. I settled on VictoriaLogs and VictoriaMetrics for the actual observing bits, fluentd for a log driver (I’m using docker-compose), and postgres-exporter, node-exporter, and handrolled metrics endpoints for prometheus metrics exposure (victoriametrics can do the metrics scraping itself).
If I were to change anything at the moment, I’d replace fluentd first, as some of the filters I use require the ruby interpreter and that seems a little heavy for my setup. Second I’d probably move to Grafana for dashboards. Alerting is gaping hole in the setup right now; I’ve used alertmanager in the past and not been particularly happy with it, but I don’t have a specific alternative I would suggest at the moment. If I had a much larger deployment, especially spread across multiple nodes, I’d consider ProxMox, but I’m just not there yet.
There were other suggestions that I didn’t go too deep into, especially after I got something working sufficiently, including dockhand, gatus and beszel, but so far (and to my surprise) having what I described above all running in containers on this little pi is doing the trick.
I’d second this. Victoria metrics and Victoria logs is orders of magnitude better than Prometheus and Loki. However, I’d switch fluent for Vector for logs collection.
Ah good to know re:vector. When I have some time I’ll give that a look.
Thanks for sharing. Some good info here.
I use the Grafana stack but I do not self host it. It’s the only thing I use in the cloud with their free offering which is pretty generous and good enough for homelabbing. I have also recently transition away from Alloy to Vector, mainly to give that a try and because the configuration of Vector seems easier and somewhat more intuitive.
I use Prometheus with various exporters pushed to Grafana which I have displayed on a small screen running on a ROCK Pi in my office
Portainer, beszel, and Whats up docker
I’ve got grafana and Prometheus, but I’m currently working on a generic log pulling script that will then feed to a local ollama instance to summarize the logs and let me know if there are any potential issues I should be aware of. Grafana is great for current state, but I’m a big believer in watching the logs to see trends, but I’m too lazy to do it by hand.
Once that’s up and running and steady, I’ll probably tie it into ntfy to actually alert me in real time
Cool, sounds like a fun project.
Not sure if you have been looking into Jev or other System 1 models, but it might be a good fit since it can output a structured type-safe output. Most of the automation I’ve tried in the past with LLMs failed due to the output structure being hallucinated (ignoring my requested structure). But it sounds like System 1 models might fix that.
I’ve been looking at an open source model called Laya a little and it might work well for your project. Sounds like you’d want structured output.
What kind of rig do you have ollama running on?
Loki?
I may eventually add that, but right now my goal isn’t the actual logging itself, it’s creating a digest and summarizing the logs down to the key points, and eventually getting a notification every morning if something weird showed up. I’m definitely going to add this to my backlog though, thanks for the suggestion
