speculative decoding in llama.cpp

[bytesupply76]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Curious whether that lines up with what other people are seeing.

[embershed99]

same here on my side.

[vectorport]

same here on my side.

That matches what I was seeing. That matches what I saw on GB200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[mangotinker73]

Did a small comparison this afternoon. On M3 Max with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[embershed99]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[mintforge]

Ran a quick check on my box. On GB200 with SGLang, I saw basically the same behavior.

[thetagrid15]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[pixelcrate82]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Not glamorous advice, but repeatable baselines save a lot of time.

[pixelcrate82]

same here on my side.

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[oxidestream]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.