+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: best small quant that still feels smart (/showthread.php?tid=691)
best small quant that still feels smart - paperrunner - 10-03-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Llama, TensorRT-LLM, M3 Max, with a lot of attention on long context.
Feel free to tear it apart if you see something dumb.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: best small quant that still feels smart - vramport - 10-04-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: best small quant that still feels smart - benchplease - 10-04-2026
that fixed it for me too.