eval setup for local models: setup advice

[batchsize]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tool calling.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[circuitworks90]

Interesting, I hadn't tried that. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cacheline]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[circuitworks90]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[routearch]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[kernelpeak]

Thanks, that helps. The pattern around prefill/decode split has been pretty similar for me.

[labline]

that fixed it for me too.