draft model choices: early impressions

[llmpeak]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and TensorRT-LLM for Mistral. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

[latentcraft]

following this because I'm in the same boat.

[cacheline]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheplane98]

That matches what I was seeing. The pattern around HBM constraints has been pretty similar for me.

[amberrunner]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[sanescope]

same here on my side.

[cacheline]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[plainember52]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorou

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. Not glamorous advice, but repeatable baselines save a lot of time.

[cacheline]

[quote]My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not g

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[batchscope72]

same here on my side.