frontier API value lately

[routecraft]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and LM Studio for Mistral. The interesting difference for me has been NVFP4.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

[routecraft]

A couple questions before I guess too hard. Are you testing this on GB200 or something else? I keep promising myself this is the last rebuild.

[thetaarch43]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[routecraft]

I tested this tonight. On M3 Max with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.

[sparrowworks88]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[routecraft]

that fixed it for me too.

[oxiderunner]

I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[routecraft]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[orbitbench6]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[routecraft]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.