model fallback rules that work

[homelabhenry]

Good point. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[phaseport36]

following this because I'm in the same boat.

[turboscope5]

Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[sparrowshed]

Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't gener

I tested this tonight. On dual 3090 with TensorRT-LLM, I saw basically the same behavior.

[macropilot]

Interesting, I hadn't tried that. That matches what I saw on dual 3090.

[amberrunner]

[quote]Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Could be a workload difference, but I wouldn'

Can you fill in a bit more context? Are you testing this on A6000 or something else?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.