speculative decoding actually worth it with LM Studio

[cacheline]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like KV cache pressure decides everything downstream.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[rocmmixer]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cacheline]

What exact setup are you running? Are you testing this on GB200 or something else?

[bytelab]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[fusedriver]

Good point. I ended up in about the same place using SGLang.

[cacheline]

Not sure I agree there. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[southtinker]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[cacheline]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup. Turns out 'just one more test' was a lie.

[orbitgarden]

I'd start with the simpler stack. Turns out 'just one more test' was a lie.

[mellowwire98]

I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.