comparing models without fooling yourself

[batchsize]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[cacheline]

I hit almost the same issue last night.

[kernelghost]

I hit almost the same issue last night.

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.