best local runtime for mixed models

[mellowwire98]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once continuous batching becomes important.

Trying to avoid rebuilding the whole thing twice.

[vectorpoint]

A couple questions before I guess too hard. Are you testing this on 5090 or something else?

[vectorhouse71]

Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[mellowwire98]

Appreciate the detail. The pattern around tokens/sec has been pretty similar for me.

[thetanest]

I'd start with the simpler stack.

[papermixer4]

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[vectorember]

Maybe, but I had the opposite result. I've seen people blame HBM constraints when the real issue was much more mundane.

[vrambandit]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.