Posts: 128
Threads: 12
Joined: Sep 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Llama, vLLM, 7900 XTX, with a lot of attention on small local stacks.
Feel free to tear it apart if you see something dumb.
rack in the garage • local models everywhere
Posts: 40
Threads: 4
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. On mini PC with llama.cpp, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Posts: 31
Threads: 5
Joined: Sep 2026
Reputation:
0
What exact setup are you running? What model, what runtime, and roughly what context size?
Posts: 128
Threads: 12
Joined: Sep 2026
Reputation:
0
Good point. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
rack in the garage • local models everywhere
Posts: 34
Threads: 4
Joined: Aug 2026
Reputation:
0
Appreciate the detail. The pattern around continuous batching has been pretty similar for me.
small models can still be good
Posts: 40
Threads: 8
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. I don't think your conclusion is far off.
Posts: 35
Threads: 7
Joined: Sep 2026
Reputation:
0
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
curious and caffeinated
Posts: 37
Threads: 5
Joined: Oct 2026
Reputation:
0
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
