best local runtime for mixed models: early impressions

[dockridge]

Did a bit of testing and thought I'd post the result here.

Short version: SGLang behaved well for me, but only after I stopped ignoring tool calling.

Happy to post more details if anyone wants them.

[fusedhall]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[vramforge]

That matches what I was seeing. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[edgeriver]

That matches what I was seeing. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[latentscope35]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[localcraft17]

I'd start with the simpler stack.

[mellowwire98]

Did a bit of testing and thought I'd post the result here.

A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[tensorkite]

What exact setup are you running? What model, what runtime, and roughly what context size?