08-29-2026, 03:11 PM
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Qwen, llama.cpp, Threadripper box, with a lot of attention on router policy.
Maybe useful as a starting point for newer folks.
This is mostly for local work rather than public serving.
The part that surprised me was how much the boring details mattered more than the model choice.
Just sharing a setup that ended up working reasonably well: Qwen, llama.cpp, Threadripper box, with a lot of attention on router policy.
Maybe useful as a starting point for newer folks.
This is mostly for local work rather than public serving.
The part that surprised me was how much the boring details mattered more than the model choice.
throughput matters, but tail latency matters too