Instinct pricing thoughts with TensorRT-LLM

[dockworks23]

I'd start with the simpler stack.

[mangohouse]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchscope72]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?