+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=65)
+--- Thread: eval setup for local models (/showthread.php?tid=382)
Pages:
1
2
eval setup for local models - benchplease - 09-21-2026
Curious what people here are seeing.
I'm mostly working with Phi in PyTorch on A6000. The part I'm trying to understand is VRAM headroom.
Would appreciate practical advice rather than theory.
Re: eval setup for local models - kernelarch - 09-22-2026
I hit almost the same issue last night. At this point the real bottleneck is my patience.
Re: eval setup for local models - cacheline - 09-22-2026
Quote:I hit almost the same issue last night. At this point the real bottleneck is my patience.
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: eval setup for local models - circuitworks90 - 09-22-2026
Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: eval setup for local models - routearch - 09-23-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: eval setup for local models - cacheline - 09-23-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quali
I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: eval setup for local models - benchplease - 09-23-2026
Quote:[quote]Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, no
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: eval setup for local models - cacheline - 09-23-2026
following this because I'm in the same boat.
Re: eval setup for local models - nanobrook - 09-23-2026
I'd start with the simpler stack.
Re: eval setup for local models - benchplease - 09-25-2026
that fixed it for me too.