+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Recipes & Configurations (https://www.datasculpting.ai/inference/forumdisplay.php?fid=52)
+---- Thread: model fallback rules that work (/showthread.php?tid=895)
model fallback rules that work - kernelrunner - 10-09-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.
Nothing scientific here, just practical notes from a real setup.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: model fallback rules that work - tensorfield88 - 10-09-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: model fallback rules that work - llm_wren - 10-10-2026
I'd start with the simpler stack.
Re: model fallback rules that work - routecraft - 10-13-2026
Quote:I'd start with the simpler stack.
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: model fallback rules that work - torchsupply99 - 10-13-2026
Good point. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: model fallback rules that work - bytepilot - 10-13-2026
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: model fallback rules that work - rocmpath - 10-15-2026
Interesting, I hadn't tried that. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: model fallback rules that work - turboyard94 - 10-18-2026
Can you fill in a bit more context? Are you testing this on Threadripper box or something else?
Re: model fallback rules that work - southtinker - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.