+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: what changed after the inference shift (/showthread.php?tid=394)
Pages:
1
2
what changed after the inference shift - tensorfield88 - 09-21-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.
Would rather learn from your scars than collect my own all at once.
Re: what changed after the inference shift - prompttinker44 - 09-22-2026
Quote:Posting this in case it saves someone else a little time.
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: what changed after the inference shift - fastfield4 - 09-23-2026
Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: what changed after the inference shift - signalcraft14 - 09-23-2026
Quote:Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more
subscribed, curious where this lands.
Re: what changed after the inference shift - vramport - 09-23-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: what changed after the inference shift - sparrowshed - 09-24-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: what changed after the inference shift - sanevault - 09-24-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: what changed after the inference shift - bytegarden73 - 09-25-2026
Quote:What exact setup are you running? What model, what runtime, and roughly what context size?
that fixed it for me too.
Re: what changed after the inference shift - southspark - 09-26-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy.
Re: what changed after the inference shift - vramport - 09-26-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.