10-07-2026, 07:19 PM
Wanted to compare notes with people doing similar things.
My current setup is TensorRT-LLM with 5090, and I keep running into continuous batching.
If you've tested something similar, what did you keep and what did you stop bothering with?
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
My current setup is TensorRT-LLM with 5090, and I keep running into continuous batching.
If you've tested something similar, what did you keep and what did you stop bothering with?
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.