speculative decoding in llama.cpp: anyone testing this?

[mellowwire98]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and MLX for Gemma. The interesting difference for me has been prefill/decode split.

Trying to avoid rebuilding the whole thing twice.

[pixelcrate82]

Appreciate the detail. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[macroforge]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[mangotinker73]

A couple questions before I guess too hard. Are you testing this on mini PC or something else?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[kernelpeak]

That matches what I was seeing. That matches what I saw on GB200.

[bytesupply76]

That matches what I was seeing. That matches what I saw on GB200.

worth checking the driver first.

[mangotinker73]

That matches what I was seeing. That matches what I saw on GB200.

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[macroforge]

Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[visionyard23]

Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack

Not sure I agree there. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

I'm happy to test one more variant if you want a second data point. At this point the real bottleneck is my patience.

[vectorport]

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.