human evals on small teams

[quantghost]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate MoE serving until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[signalcraft14]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[torchshed]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[oxiderunner]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[turboyard94]

Ran a quick check on my box. On M3 Max with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.

[audiodesk]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[visionforge]

Ran a quick check on my box. On B200 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[cozysmith]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.

[lablab]

worth checking the driver first.