speculative decoding in llama.cpp with LM Studio

[vrambandit]

following this because I'm in the same boat.

[kernelgrid]

That matches what I was seeing. That matches what I saw on 7900 XTX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vrambandit]

same here on my side.

[cudaline]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[mintquill]

Good point. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mangotinker73]

Good point. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the wor

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.