+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=65)
+--- Thread: eval setup for local models: setup advice (/showthread.php?tid=513)
eval setup for local models: setup advice - batchsize - 09-27-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tool calling.
Would rather learn from your scars than collect my own all at once.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: eval setup for local models: setup advice - circuitworks90 - 09-27-2026
Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: eval setup for local models: setup advice - cacheline - 09-30-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: eval setup for local models: setup advice - circuitworks90 - 09-30-2026
Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: eval setup for local models: setup advice - routearch - 09-30-2026
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: eval setup for local models: setup advice - kernelpeak - 09-30-2026
Thanks, that helps. The pattern around prefill/decode split has been pretty similar for me.
Re: eval setup for local models: setup advice - labline - 10-01-2026
that fixed it for me too.