+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distill or just quantize (/showthread.php?tid=554)
Pages:
1
2
distill or just quantize - quiettoken - 09-28-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mistral, LM Studio, 7900 XTX, with a lot of attention on VRAM headroom.
If someone wants the exact settings I can clean them up and post them.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: distill or just quantize - quantshed - 09-30-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.
Re: distill or just quantize - cozykite - 09-30-2026
Quote:I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: distill or just quantize - oxidegrove40 - 09-30-2026
Good point. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: distill or just quantize - quietstack - 10-03-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: distill or just quantize - quantshed - 10-03-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: distill or just quantize - torchtrack24 - 10-03-2026
Good point. I don't think your conclusion is far off.
Re: distill or just quantize - batchscope72 - 10-06-2026
Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane.
Re: distill or just quantize - fusedriver - 10-06-2026
Quote:Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane.
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.
Re: distill or just quantize - quantshed - 10-08-2026
I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.