distill or just quantize

[quiettoken]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, LM Studio, 7900 XTX, with a lot of attention on VRAM headroom.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[quantshed]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[cozykite]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

If you post your exact settings, people here can probably narrow it down pretty fast.

[oxidegrove40]

Good point. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[quietstack]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[quantshed]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[torchtrack24]

Good point. I don't think your conclusion is far off.

[batchscope72]

Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane.

[fusedriver]

Not sure I agree there. I've seen people blame small local stacks when the real issue was much more mundane.

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.

[quantshed]

I'd approach it in a boring order. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.