+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion with PyTorch (/showthread.php?tid=713)
Pages:
1
2
Search / Retrieval / Reranking discussion with PyTorch - cacheline - 10-03-2026
Maybe I'm missing something obvious.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: Search / Retrieval / Reranking discussion with PyTorch - fastscope - 10-04-2026
That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Search / Retrieval / Reranking discussion with PyTorch - vramarch - 10-06-2026
That matches what I was seeing. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Search / Retrieval / Reranking discussion with PyTorch - fusedhall56 - 10-06-2026
same here on my side.
Re: Search / Retrieval / Reranking discussion with PyTorch - tensorport - 10-07-2026
Appreciate the detail. The pattern around tokens/sec has been pretty similar for me.
Re: Search / Retrieval / Reranking discussion with PyTorch - macrostack69 - 10-10-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Search / Retrieval / Reranking discussion with PyTorch - ggufwire22 - 10-10-2026
What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: Search / Retrieval / Reranking discussion with PyTorch - kernelmike - 10-10-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: Search / Retrieval / Reranking discussion with PyTorch - rocmriley - 10-11-2026
Quote:What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
I'd start with the simpler stack.
Re: Search / Retrieval / Reranking discussion with PyTorch - cachebench - 10-11-2026
Appreciate the detail. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.