model fallback rules that work

[dockember]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Curious whether that lines up with what other people are seeing.

[kernelmike]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[llmarch51]

Good point. That matches what I saw on GB200.

[sparrowhall]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[cozywire]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling.

[kernelpeak]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure.

[batchshed]

Not sure I agree there. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere.

[vectorpoint]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[cacheplane98]

Good point. I ended up in about the same place using TensorRT-LLM.

[batchsize]

Good point. I ended up in about the same place using TensorRT-LLM.

Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.