model fallback rules that work

[batchsize]

Good point. I ended up in about the same place using TensorRT-LLM.

Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.