+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Ollama / LM Studio / Desktop Runtimes (https://www.datasculpting.ai/inference/forumdisplay.php?fid=38)
+--- Thread: best local runtime for mixed models for Gemma (/showthread.php?tid=1001)
Pages:
1
2
best local runtime for mixed models for Gemma - fusedhall - 10-11-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.
I'm especially interested in mistakes worth avoiding.
This is mostly for local work rather than public serving.
Re: best local runtime for mixed models for Gemma - amberhall - 10-11-2026
What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best local runtime for mixed models for Gemma - vectorhouse71 - 10-11-2026
I hit almost the same issue last night.
Re: best local runtime for mixed models for Gemma - latentscope35 - 10-11-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: best local runtime for mixed models for Gemma - fusedhall - 10-11-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: best local runtime for mixed models for Gemma - mellowwire98 - 10-11-2026
Quote:Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Good point. That matches what I saw on M3 Max.
Re: best local runtime for mixed models for Gemma - amberhall - 10-12-2026
I don't think that's the whole story. I've seen people blame tool calling when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: best local runtime for mixed models for Gemma - vectorember - 10-12-2026
I hit almost the same issue last night.
Re: best local runtime for mixed models for Gemma - quantshed - 10-12-2026
Interesting, I hadn't tried that. That matches what I saw on GB200.
Re: best local runtime for mixed models for Gemma - vectorember - 10-12-2026
Quote:Interesting, I hadn't tried that. That matches what I saw on GB200.
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.