recipes for a good local stack

[llmpeak]

Did a small comparison this afternoon. On 5090 with SGLang, I saw basically the same behavior.

[audionode8]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[cachebench]

Good point. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[thetagrid15]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[marbleyard32]

Appreciate the detail. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.