+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Ollama / LM Studio / Desktop Runtimes (https://www.datasculpting.ai/inference/forumdisplay.php?fid=38)
+--- Thread: desktop runtimes for non-devs (/showthread.php?tid=89)
desktop runtimes for non-devs - prompttinker44 - 08-31-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: SGLang on mini PC, keeping continuous batching in mind from the start.
Maybe useful as a starting point for newer folks.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: desktop runtimes for non-devs - batchscope72 - 08-31-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: desktop runtimes for non-devs - latentscope35 - 09-01-2026
I tested this tonight. On GB200 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: desktop runtimes for non-devs - batchfield55 - 09-02-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: desktop runtimes for non-devs - oxidemixer - 09-02-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: desktop runtimes for non-devs - orbitbench6 - 09-02-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: desktop runtimes for non-devs - localcraft17 - 09-02-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: desktop runtimes for non-devs - amberhall - 09-02-2026
Did a small comparison this afternoon. On mini PC with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: desktop runtimes for non-devs - visionforge - 09-02-2026
My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.