+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Edge AI & Gadgets (https://www.datasculpting.ai/inference/forumdisplay.php?fid=21)
+--- Forum: SBCs / Mini PCs (https://www.datasculpting.ai/inference/forumdisplay.php?fid=74)
+--- Thread: SBC inference experiments for DeepSeek (/showthread.php?tid=280)
Pages:
1
2
SBC inference experiments for DeepSeek - promptfield - 09-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Kimi, Ollama, B200, with a lot of attention on quant drift.
Feel free to tear it apart if you see something dumb.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: SBC inference experiments for DeepSeek - vectorport - 09-15-2026
I think the bottleneck is somewhere else. I've seen people blame long context when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: SBC inference experiments for DeepSeek - cobalttinker - 09-15-2026
Quote:I think the bottleneck is somewhere else. I've seen people blame long context when the real issue was much more mundane. Worth testing the simpler exp
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: SBC inference experiments for DeepSeek - mangopeak72 - 09-15-2026
Thanks, that helps. I don't think your conclusion is far off.
Re: SBC inference experiments for DeepSeek - cobalttinker - 09-15-2026
Quote:Thanks, that helps. I don't think your conclusion is far off.
that fixed it for me too.
Re: SBC inference experiments for DeepSeek - routecrate - 09-15-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.
Re: SBC inference experiments for DeepSeek - homelabhenry - 09-15-2026
that fixed it for me too.
Re: SBC inference experiments for DeepSeek - vramghost - 09-15-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: SBC inference experiments for DeepSeek - phasespark - 09-15-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.
Re: SBC inference experiments for DeepSeek - prompttinker44 - 09-15-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once
Good point. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.