+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Ollama / LM Studio / Desktop Runtimes (https://www.datasculpting.ai/inference/forumdisplay.php?fid=38)
+--- Thread: best local runtime for mixed models (/showthread.php?tid=1289)
Pages:
1
2
best local runtime for mixed models - vectorember - 10-15-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Simple and boring is a valid answer.
Re: best local runtime for mixed models - oxidemixer - 10-17-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: best local runtime for mixed models - edgeriver - 10-18-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: best local runtime for mixed models - edgeriver - 10-18-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: best local runtime for mixed models - vectorhouse71 - 10-18-2026
What worked for me was pretty simple. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best local runtime for mixed models - edgeriver - 10-18-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: best local runtime for mixed models - ggufwire22 - 10-18-2026
I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: best local runtime for mixed models - pixelspark - 10-18-2026
Tried to reproduce it just now. On Threadripper box with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: best local runtime for mixed models - edgeriver - 10-18-2026
My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: best local runtime for mixed models - cacheworks - 10-18-2026
Quote:My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.