About 200 t/s prompt processing and 10-20 t/s with MTP.
Greatly depends on the task, predictable things like code generates at 18-20 t/s. Creative writing more like 10-17 t/s.
- 0 Posts
- 4 Comments
Joined 3 years ago
Cake day: June 4th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.
Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.
I’m running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.
I use them for:
- Tagging and classification of my documents in Paperless
- Home Assistant (voice assistant)
- Translations (both text and image)
- Transcriptions
- Some light coding and debugging
- Avatar/Backdrop generation for DnD sessions
If you have trouble with outgoing mails, you can use a hybrid approach.
Receive mails directly to your server but use a mail service to relay your outgoing mails. Configuration for that is very simple in mailcow and there are a few dozen (free) transactional email providers (e.g. Scaleway).
That way you can keep receiving your mails privately and only have to give up some privacy when sending mails.
Q8 from unsloth.
My go to model for knowledge. Definitely much faster at Q5 but it lacks the tool calling quality of the Qwen3.6 models. Really hoping we see a Qwen3.6-122b soon…