+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: RAG (https://www.datasculpting.ai/inference/forumdisplay.php?fid=58)
+--- Thread: where reranking mattered on mini PC (/showthread.php?tid=446)
Pages:
1
2
where reranking mattered on mini PC - oxideworks59 - 09-24-2026
Wanted to compare notes with people doing similar things.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling structured output.
Would appreciate practical advice rather than theory. The logs were somehow calmer than I was.
Re: where reranking mattered on mini PC - turbogrid11 - 09-24-2026
Quote:Wanted to compare notes with people doing similar things.
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
Re: where reranking mattered on mini PC - routecrate - 09-24-2026
Appreciate the detail. The pattern around quant drift has been pretty similar for me.
Re: where reranking mattered on mini PC - sparrowbrook - 09-24-2026
same here on my side.
Re: where reranking mattered on mini PC - audionode8 - 09-24-2026
Good point. I don't think your conclusion is far off.
Re: where reranking mattered on mini PC - tinycontext - 09-24-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: where reranking mattered on mini PC - audionode8 - 09-24-2026
Thanks, that helps. The pattern around HBM constraints has been pretty similar for me.
Re: where reranking mattered on mini PC - routecrate - 09-25-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: where reranking mattered on mini PC - sanepeak48 - 09-25-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: where reranking mattered on mini PC - audionode8 - 09-25-2026
Ran a quick check on my box. On 4090 with SGLang, I saw basically the same behavior.