+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion with PyTorch (/showthread.php?tid=713)
Pages:
1
2
Re: Search / Retrieval / Reranking discussion with PyTorch - phasespark - 10-11-2026
Quote:Appreciate the detail. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at w
Ran a quick check on my box. On M3 Max with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: Search / Retrieval / Reranking discussion with PyTorch - batchsize - 10-12-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: Search / Retrieval / Reranking discussion with PyTorch - llmarch83 - 10-15-2026
Quote:My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of the
I hit almost the same issue last night.
Re: Search / Retrieval / Reranking discussion with PyTorch - mangoquill - 10-15-2026
Good point. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Search / Retrieval / Reranking discussion with PyTorch - cudascope - 10-16-2026
Interesting, I hadn't tried that. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.