benchmarks that map to real use with vLLM

[thetanest]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[batchsize]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[quantghost]

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[northcraft90]

Tried to reproduce it just now. On Threadripper box with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.

[ggufwire22]

Tried to reproduce it just now. On Threadripper box with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not qualit

following this because I'm in the same boat.

[sanevault]

Maybe, but I had the opposite result. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.