10-11-2026, 01:35 AM
Spent some time on this today and wanted to compare notes.
I've gone back and forth between vLLM and TensorRT-LLM for Gemma. The interesting difference for me has been batch size.
What would you pick if you had to live with it for a few months?
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.
I've gone back and forth between vLLM and TensorRT-LLM for Gemma. The interesting difference for me has been batch size.
What would you pick if you had to live with it for a few months?
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. At this point the real bottleneck is my patience.
still benchmarking everything