10-06-2026, 05:33 AM
Spent some time on this today and wanted to compare notes.
I've gone back and forth between TensorRT-LLM and SGLang for Mixtral. The interesting difference for me has been tail latency.
Trying to avoid rebuilding the whole thing twice.
I've gone back and forth between TensorRT-LLM and SGLang for Mixtral. The interesting difference for me has been tail latency.
Trying to avoid rebuilding the whole thing twice.