09-06-2026, 07:17 AM
I'm trying to sanity-check my setup.
My current setup is TensorRT-LLM with 4090, and I keep running into continuous batching.
If you've tested something similar, what did you keep and what did you stop bothering with?
This is mostly for local work rather than public serving.
My current setup is TensorRT-LLM with 4090, and I keep running into continuous batching.
If you've tested something similar, what did you keep and what did you stop bothering with?
This is mostly for local work rather than public serving.
measure twice, benchmark three times