+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Quantization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=40)
+--- Thread: best small quant that still feels smart on 4090 (/showthread.php?tid=1156)
Pages:
1
2
best small quant that still feels smart on 4090 - vramscope - 10-14-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once token cost becomes important.
What would you pick if you had to live with it for a few months?
Re: best small quant that still feels smart on 4090 - cacheworks - 10-14-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: best small quant that still feels smart on 4090 - nanobrook - 10-14-2026
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: best small quant that still feels smart on 4090 - mangofield33 - 10-14-2026
subscribed, curious where this lands.
Re: best small quant that still feels smart on 4090 - phasestream53 - 10-15-2026
What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: best small quant that still feels smart on 4090 - benchplease - 10-15-2026
that fixed it for me too.
Re: best small quant that still feels smart on 4090 - phasestream53 - 10-15-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: best small quant that still feels smart on 4090 - benchplease - 10-16-2026
Quote:I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads e
Tried to reproduce it just now. On B200 with vLLM, I saw basically the same behavior.
Re: best small quant that still feels smart on 4090 - rocmhouse90 - 10-16-2026
Quote:[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these th
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
I'm happy to test one more variant if you want a second data point.
Re: best small quant that still feels smart on 4090 - cudaline - 10-17-2026
I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.