where reranking mattered on mini PC

[oxideworks59]

Wanted to compare notes with people doing similar things.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling structured output.

Would appreciate practical advice rather than theory. The logs were somehow calmer than I was.

[turbogrid11]

Wanted to compare notes with people doing similar things.

Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[routecrate]

Appreciate the detail. The pattern around quant drift has been pretty similar for me.

[sparrowbrook]

same here on my side.

[audionode8]

Good point. I don't think your conclusion is far off.

[tinycontext]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[audionode8]

Thanks, that helps. The pattern around HBM constraints has been pretty similar for me.

[routecrate]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[sanepeak48]

What exact setup are you running? Is this local-only or are you routing anything externally?

[audionode8]

Ran a quick check on my box. On 4090 with SGLang, I saw basically the same behavior.