+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion (/showthread.php?tid=875)
Pages:
1
2
Search / Retrieval / Reranking discussion - signalcraft14 - 10-07-2026
I'm trying to sanity-check my setup.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling continuous batching.
Would appreciate practical advice rather than theory.
This is mostly for local work rather than public serving.
Re: Search / Retrieval / Reranking discussion - cobaltlab - 10-07-2026
I'd start with the simpler stack.
Re: Search / Retrieval / Reranking discussion - macrotinker - 10-07-2026
Quote:I'd start with the simpler stack.
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: Search / Retrieval / Reranking discussion - fusedhall56 - 10-07-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: Search / Retrieval / Reranking discussion - signalcraft14 - 10-07-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: Search / Retrieval / Reranking discussion - batchshed - 10-07-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift.
Re: Search / Retrieval / Reranking discussion - dockworks23 - 10-08-2026
I'd approach it in a boring order. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Search / Retrieval / Reranking discussion - audiodesk - 10-11-2026
subscribed, curious where this lands.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: Search / Retrieval / Reranking discussion - southpatch - 10-13-2026
that fixed it for me too.
Re: Search / Retrieval / Reranking discussion - tensorfield88 - 10-13-2026
A couple things I'd check first. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.