desktop runtimes for non-devs

[prompttinker44]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: SGLang on mini PC, keeping continuous batching in mind from the start.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[batchscope72]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[latentscope35]

I tested this tonight. On GB200 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[batchfield55]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[oxidemixer]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[orbitbench6]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[localcraft17]

Need a little more detail here. Is this local-only or are you routing anything externally?

[amberhall]

Did a small comparison this afternoon. On mini PC with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[visionforge]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.