+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Workflows / Automation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=47)
+--- Thread: frontier API value lately (/showthread.php?tid=87)
Pages:
1
2
frontier API value lately - routecraft - 09-01-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between llama.cpp and LM Studio for Mistral. The interesting difference for me has been NVFP4.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: frontier API value lately - routecraft - 09-01-2026
A couple questions before I guess too hard. Are you testing this on GB200 or something else? I keep promising myself this is the last rebuild.
Re: frontier API value lately - thetaarch43 - 09-01-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: frontier API value lately - routecraft - 09-01-2026
I tested this tonight. On M3 Max with SGLang, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: frontier API value lately - sparrowworks88 - 09-02-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: frontier API value lately - routecraft - 09-05-2026
that fixed it for me too.
Re: frontier API value lately - oxiderunner - 09-06-2026
I think the bottleneck is somewhere else. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: frontier API value lately - routecraft - 09-07-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: frontier API value lately - orbitbench6 - 09-10-2026
Quote:My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end
Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: frontier API value lately - routecraft - 09-10-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.