+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG recall vs speed tradeoff (/showthread.php?tid=141)
Pages:
1
2
RAG recall vs speed tradeoff - cobaltworks25 - 09-05-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like router policy decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: RAG recall vs speed tradeoff - homelabhenry - 09-07-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: RAG recall vs speed tradeoff - llmarch51 - 09-08-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: RAG recall vs speed tradeoff - mellowsupply - 09-08-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: RAG recall vs speed tradeoff - phasestream53 - 09-08-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: RAG recall vs speed tradeoff - localpatch78 - 09-08-2026
Quote:I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads e
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: RAG recall vs speed tradeoff - bytepilot - 09-09-2026
Quote:[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these th
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: RAG recall vs speed tradeoff - turboscope - 09-10-2026
Quote:[quote][quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of t
that fixed it for me too.
Re: RAG recall vs speed tradeoff - rocmmixer - 09-10-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Not glamorous advice, but repeatable baselines save a lot of time.
Re: RAG recall vs speed tradeoff - batchsize - 09-10-2026
Appreciate the detail. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.