frontier API value lately with Ollama

[routecraft]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between SGLang and SGLang for Kimi. The interesting difference for me has been HBM constraints.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[batchsize]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[routecraft]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Once you know the plain path works, then add the faster path back in piece by piece.

[routecraft]

Thanks, that helps. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[dockworks23]

Thanks, that helps. That matches what I saw on dual 3090. Worth rechecking your defaults though, because one bad assumption can make the rest of the t

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[routecraft]

I think the bottleneck is somewhere else. I've seen people blame NVFP4 when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[sparrowbrook]

My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[routecraft]

Appreciate the detail. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[batchember]

subscribed, curious where this lands.

[routecraft]

A couple questions before I guess too hard. Are you testing this on A6000 or something else?