what are you all running this week on 5090

[rocmplane]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate tail latency until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

[sanepeak48]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[cacheplane98]

A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[torchkite]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

I'm happy to test one more variant if you want a second data point.

[rocmrunner40]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.

If you post your exact settings, people here can probably narrow it down pretty fast.

[latentscope35]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[tensorkite]

Good point. That matches what I saw on A6000.

[quiettoken]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[thetagrid15]

A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmplane]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.