model fallback rules that work

[quietcraft]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[quantyard]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[mlxgarden]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[rocmrunner40]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

I tested this tonight. On 7900 XTX with SGLang, I saw basically the same behavior. I expected the model choice to matter more than it did.

[routeline23]

following this because I'm in the same boat.

If you post your exact settings, people here can probably narrow it down pretty fast.

[southden]

Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cozyarch]

Appreciate the detail. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make th

I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[quantshed]

Thanks, that helps. I ended up in about the same place using llama.cpp.

[relaynest82]

What exact setup are you running? Is this local-only or are you routing anything externally?

[batchcraft]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Not glamorous advice, but repeatable baselines save a lot of time.