SBC inference experiments for DeepSeek

[promptfield]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Kimi, Ollama, B200, with a lot of attention on quant drift.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

[vectorport]

I think the bottleneck is somewhere else. I've seen people blame long context when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[cobalttinker]

I think the bottleneck is somewhere else. I've seen people blame long context when the real issue was much more mundane. Worth testing the simpler exp

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[mangopeak72]

Thanks, that helps. I don't think your conclusion is far off.

[cobalttinker]

Thanks, that helps. I don't think your conclusion is far off.

that fixed it for me too.

[routecrate]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching.

[homelabhenry]

that fixed it for me too.

[vramghost]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[phasespark]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once you know the plain path works, then add the faster path back in piece by piece.

[prompttinker44]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. Once

Good point. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.