eval setup for local models

[circuitworks90]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[griffinloom66]

Tried to reproduce it just now. On GB200 with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.

[nanobrook]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelarch]

same here on my side.

[northgrid33]

following this because I'm in the same boat.

[benchplease]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[quiethouse]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[mangowire]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up bef

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.