+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: RAG (https://www.datasculpting.ai/inference/forumdisplay.php?fid=58)
+--- Thread: where reranking mattered on mini PC (/showthread.php?tid=1205)
Pages:
1
2
where reranking mattered on mini PC - orbitgarden - 10-14-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.
Would rather learn from your scars than collect my own all at once.
Re: where reranking mattered on mini PC - labpath - 10-16-2026
A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: where reranking mattered on mini PC - batchshed - 10-18-2026
Quote:That matches what I was seeing. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at
Thanks, that helps. The pattern around batch size has been pretty similar for me.
Re: where reranking mattered on mini PC - audionode8 - 10-18-2026
I hit almost the same issue last night.
Re: where reranking mattered on mini PC - edgequill - 10-18-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: where reranking mattered on mini PC - kernelghost - 10-18-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: where reranking mattered on mini PC - orbitline - 10-18-2026
I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: where reranking mattered on mini PC - quietsupply11 - 10-18-2026
Quote:A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Not glamorous advice, but repeatable baselines save a lot of time.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: where reranking mattered on mini PC - deltastation - 10-18-2026
Did a small comparison this afternoon. On A6000 with SGLang, I saw basically the same behavior.
Re: where reranking mattered on mini PC - mlxgarden - 10-19-2026
My first guess would be this: On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Turns out 'just one more test' was a lie.