what truly stays local with LM Studio

[vramstream]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, PyTorch, 4090, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.

[mintstation99]

Thought I'd post a small recipe because a lot of threads ask similar things.

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[audionode8]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[audiopath66]

Good point. The pattern around speculative decoding has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[audioriver]

that fixed it for me too.

[mintstation99]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cobaltspark]

that fixed it for me too.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[relayforge]

What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmriley]

that fixed it for me too.

[mlxgarden]

Thanks, that helps. The pattern around workstation thermals has been pretty similar for me.