+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Context / KV Cache (https://www.datasculpting.ai/inference/forumdisplay.php?fid=41)
+--- Thread: draft model choices: early impressions (/showthread.php?tid=774)
Pages:
1
2
draft model choices: early impressions - llmpeak - 10-05-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between PyTorch and TensorRT-LLM for Mistral. The interesting difference for me has been speculative decoding.
Trying to avoid rebuilding the whole thing twice.
Re: draft model choices: early impressions - latentcraft - 10-06-2026
following this because I'm in the same boat.
Re: draft model choices: early impressions - cacheline - 10-06-2026
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: draft model choices: early impressions - cacheplane98 - 10-06-2026
That matches what I was seeing. The pattern around HBM constraints has been pretty similar for me.
Re: draft model choices: early impressions - amberrunner - 10-06-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: draft model choices: early impressions - sanescope - 10-07-2026
same here on my side.
Re: draft model choices: early impressions - cacheline - 10-08-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: draft model choices: early impressions - plainember52 - 10-08-2026
Quote:My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorou
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. Not glamorous advice, but repeatable baselines save a lot of time.
Re: draft model choices: early impressions - cacheline - 10-08-2026
Quote:[quote]My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not g
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: draft model choices: early impressions - batchscope72 - 10-08-2026
same here on my side.