benchmarks that map to real use with vLLM

[sparrowcraft]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using vLLM on B200. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.

[stackarch]

Did a bit of testing and thought I'd post the result here.

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[cudaline]

same here on my side.

[oxiderunner]

Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[plainscope50]

Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the re

What exact setup are you running? Is this local-only or are you routing anything externally?

[southtinker]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[thetaplane]

Good point. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[marbletrack]

subscribed, curious where this lands.

[byteyard]

worth checking the driver first.

[orbitgarden]

that fixed it for me too.