+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=65)
+--- Thread: eval setup for local models (/showthread.php?tid=233)
eval setup for local models - circuitworks90 - 09-11-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.
Maybe obvious to some of you, but it clicked for me this week.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: eval setup for local models - griffinloom66 - 09-11-2026
Tried to reproduce it just now. On GB200 with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: eval setup for local models - nanobrook - 09-12-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: eval setup for local models - kernelarch - 09-12-2026
same here on my side.
Re: eval setup for local models - northgrid33 - 09-13-2026
following this because I'm in the same boat.
Re: eval setup for local models - benchplease - 09-13-2026
I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: eval setup for local models - quiethouse - 09-14-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: eval setup for local models - mangowire - 09-14-2026
Quote:Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up bef
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.