4090 vs 5090 for 70B quants with TensorRT-LLM

[torchshed]

Need a little more detail here. What model, what runtime, and roughly what context size?

[fusedhall56]

Appreciate the detail. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchwire]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly.