latency vs throughput for small deployments

[kernelrunner]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and GB200 felt surprisingly usable once I tuned around continuous batching.

Maybe useful as a starting point for newer folks.

[papermixer4]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[signalcraft14]

I'd start with the simpler stack.

[papermixer4]

Maybe, but I had the opposite result. I've seen people blame token cost when the real issue was much more mundane.

[bytestack]

Maybe, but I had the opposite result. I've seen people blame token cost when the real issue was much more mundane.

What worked for me was pretty simple. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[rocmpath]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[cudahouse]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cudapeak]

I hit almost the same issue last night.

[stackarch]

following this because I'm in the same boat. I keep promising myself this is the last rebuild.

[macropilot]

Good point. I don't think your conclusion is far off. The logs were somehow calmer than I was.