speculative decoding actually worth it for Qwen

[cacheline]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and 5090 felt surprisingly usable once I tuned around token cost.

Feel free to tear it apart if you see something dumb.

[cacheline]

Thought I'd post a small recipe because a lot of threads ask similar things.

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[mangopath4]

Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one. At this point the real bottleneck is my patience.

[cacheline]

Good point. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

Thanks, that helps. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[quietcraft]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[cacheline]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, no

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[sparrowkite53]

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[amberkite]

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

That matches what I was seeing. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

that fixed it for me too.