continuous batching wins and losses

[routearch]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

[amberplane]

Posting this in case it saves someone else a little time.

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.

[oxidemixer]

Interesting, I hadn't tried that. The pattern around consumer GPUs has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

Interesting, I hadn't tried that. The pattern around consumer GPUs has been pretty similar for me. Worth rechecking your defaults though, because one

worth checking the driver first.

[audioforge]

Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[oxidemixer]

I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[deltapoint17]

subscribed, curious where this lands.

[rocmrunner40]

subscribed, curious where this lands.

[signalpeak]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?