+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Ollama / LM Studio / Desktop Runtimes (https://www.datasculpting.ai/inference/forumdisplay.php?fid=38)
+--- Thread: best local runtime for mixed models (/showthread.php?tid=621)
best local runtime for mixed models - mellowwire98 - 09-30-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once continuous batching becomes important.
Trying to avoid rebuilding the whole thing twice.
Re: best local runtime for mixed models - vectorpoint - 10-01-2026
A couple questions before I guess too hard. Are you testing this on 5090 or something else?
Re: best local runtime for mixed models - vectorhouse71 - 10-01-2026
Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: best local runtime for mixed models - mellowwire98 - 10-01-2026
Appreciate the detail. The pattern around tokens/sec has been pretty similar for me.
Re: best local runtime for mixed models - thetanest - 10-02-2026
I'd start with the simpler stack.
Re: best local runtime for mixed models - papermixer4 - 10-02-2026
A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: best local runtime for mixed models - vectorember - 10-02-2026
Maybe, but I had the opposite result. I've seen people blame HBM constraints when the real issue was much more mundane.
Re: best local runtime for mixed models - vrambandit - 10-03-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.