+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distill or just quantize for DeepSeek (/showthread.php?tid=37)
Pages:
1
2
distill or just quantize for DeepSeek - turboyard94 - 08-27-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Phi using PyTorch on GB200. The headline result was better than I expected, but small local stacks still ended up being the limiting factor.
Nothing scientific here, just practical notes from a real setup.
This is mostly for local work rather than public serving.
Re: distill or just quantize for DeepSeek - cobaltlab - 08-28-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.
Re: distill or just quantize for DeepSeek - batchscope72 - 08-28-2026
A couple things I'd check first. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: distill or just quantize for DeepSeek - northquill - 08-28-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: distill or just quantize for DeepSeek - llmtrack26 - 08-28-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: distill or just quantize for DeepSeek - turboyard94 - 08-28-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: distill or just quantize for DeepSeek - dockember - 08-29-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.
Re: distill or just quantize for DeepSeek - vrampilot - 08-30-2026
I think the bottleneck is somewhere else. I get why that seems right, but on B200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: distill or just quantize for DeepSeek - emberpeak - 08-30-2026
Good point. I don't think your conclusion is far off.
Re: distill or just quantize for DeepSeek - thetagrid15 - 08-30-2026
Quote:Good point. I don't think your conclusion is far off.
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Once you know the plain path works, then add the faster path back in piece by piece.