distill or just quantize for DeepSeek

[turboyard94]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using PyTorch on GB200. The headline result was better than I expected, but small local stacks still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

[cobaltlab]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.

[batchscope72]

A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.

[northquill]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[llmtrack26]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[turboyard94]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[dockember]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.

[vrampilot]

I think the bottleneck is somewhere else. I get why that seems right, but on B200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[emberpeak]

Good point. I don't think your conclusion is far off.

[thetagrid15]

Good point. I don't think your conclusion is far off.

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Once you know the plain path works, then add the faster path back in piece by piece.