+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: best small quant that still feels smart (/showthread.php?tid=211)
Pages:
1
2
best small quant that still feels smart - deltagrid - 09-10-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: TensorRT-LLM on A6000, keeping structured output in mind from the start.
Maybe useful as a starting point for newer folks.
Re: best small quant that still feels smart - promptfield - 09-11-2026
What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: best small quant that still feels smart - relaynest82 - 09-11-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: best small quant that still feels smart - nanobrook - 09-11-2026
that fixed it for me too.
Re: best small quant that still feels smart - fastbench - 09-11-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best small quant that still feels smart - paperrunner - 09-11-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know
subscribed, curious where this lands.
Re: best small quant that still feels smart - cudaline - 09-11-2026
subscribed, curious where this lands.
Re: best small quant that still feels smart - vrambandit - 09-11-2026
Quote:subscribed, curious where this lands.
That matches what I was seeing. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: best small quant that still feels smart - cudaline - 09-11-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: best small quant that still feels smart - mangopath4 - 09-13-2026
worth checking the driver first.