where reranking mattered on mini PC

[orbitgarden]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Would rather learn from your scars than collect my own all at once.

[labpath]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[batchshed]

That matches what I was seeing. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at

Thanks, that helps. The pattern around batch size has been pretty similar for me.

[audionode8]

I hit almost the same issue last night.

[edgequill]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelghost]

Need a little more detail here. Is this local-only or are you routing anything externally?

[orbitline]

I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[quietsupply11]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Not glamorous advice, but repeatable baselines save a lot of time.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[deltastation]

Did a small comparison this afternoon. On A6000 with SGLang, I saw basically the same behavior.

[mlxgarden]

My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Turns out 'just one more test' was a lie.