+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: TensorRT-LLM (https://www.datasculpting.ai/inference/forumdisplay.php?fid=37)
+--- Thread: quant support gaps: what are you seeing? (/showthread.php?tid=1302)
Pages:
1
2
quant support gaps: what are you seeing? - batchcraft - 10-16-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.
Interested in tradeoffs more than fan loyalty.
Re: quant support gaps: what are you seeing? - syshouse - 10-16-2026
Appreciate the detail. I don't think your conclusion is far off.
Re: quant support gaps: what are you seeing? - kernelmike - 10-18-2026
Quote:A couple questions before I guess too hard. Are you testing this on B200 or something else?
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: quant support gaps: what are you seeing? - kernelgrid - 10-18-2026
I think the bottleneck is somewhere else. I've seen people blame reranking when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: quant support gaps: what are you seeing? - kernelmike - 10-19-2026
Quote:Can you fill in a bit more context? Are you testing this on 5090 or something else?
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.
Re: quant support gaps: what are you seeing? - kernelmike - 10-19-2026
Quote:Appreciate the detail. I don't think your conclusion is far off.
following this because I'm in the same boat.
Re: quant support gaps: what are you seeing? - batchcraft - 10-19-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: quant support gaps: what are you seeing? - northcraft90 - 10-19-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: quant support gaps: what are you seeing? - rasterbench - 10-19-2026
I hit almost the same issue last night.
Re: quant support gaps: what are you seeing? - mangotinker73 - 10-19-2026
Can you fill in a bit more context? Are you testing this on 5090 or something else?
If you post your exact settings, people here can probably narrow it down pretty fast.