+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: benchmarks that map to real use with vLLM (/showthread.php?tid=126)
Pages:
1
2
benchmarks that map to real use with vLLM - sparrowcraft - 09-03-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Phi using vLLM on B200. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.
Curious if anyone else landed in the same place.
Re: benchmarks that map to real use with vLLM - stackarch - 09-03-2026
Quote:Did a bit of testing and thought I'd post the result here.
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: benchmarks that map to real use with vLLM - cudaline - 09-04-2026
same here on my side.
Re: benchmarks that map to real use with vLLM - oxiderunner - 09-05-2026
Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: benchmarks that map to real use with vLLM - plainscope50 - 09-05-2026
Quote:Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the re
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: benchmarks that map to real use with vLLM - southtinker - 09-05-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: benchmarks that map to real use with vLLM - thetaplane - 09-06-2026
Good point. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: benchmarks that map to real use with vLLM - marbletrack - 09-06-2026
subscribed, curious where this lands.
Re: benchmarks that map to real use with vLLM - byteyard - 09-06-2026
worth checking the driver first.
Re: benchmarks that map to real use with vLLM - orbitgarden - 09-06-2026
that fixed it for me too.