+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Ollama / LM Studio / Desktop Runtimes (https://www.datasculpting.ai/inference/forumdisplay.php?fid=38)
+--- Thread: best local runtime for mixed models: early impressions (/showthread.php?tid=1403)
best local runtime for mixed models: early impressions - dockridge - 10-17-2026
Did a bit of testing and thought I'd post the result here.
Short version: SGLang behaved well for me, but only after I stopped ignoring tool calling.
Happy to post more details if anyone wants them.
Re: best local runtime for mixed models: early impressions - fusedhall - 10-18-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: best local runtime for mixed models: early impressions - vramforge - 10-18-2026
That matches what I was seeing. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: best local runtime for mixed models: early impressions - edgeriver - 10-18-2026
That matches what I was seeing. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: best local runtime for mixed models: early impressions - latentscope35 - 10-19-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: best local runtime for mixed models: early impressions - localcraft17 - 10-19-2026
I'd start with the simpler stack.
Re: best local runtime for mixed models: early impressions - mellowwire98 - 10-19-2026
Quote:Did a bit of testing and thought I'd post the result here.
A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: best local runtime for mixed models: early impressions - tensorkite - 10-19-2026
What exact setup are you running? What model, what runtime, and roughly what context size?