RAG recall vs speed tradeoff

[cobaltworks25]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like router policy decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[homelabhenry]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmarch51]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[mellowsupply]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[phasestream53]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[localpatch78]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads e

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[bytepilot]

[quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these th

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[turboscope]

[quote][quote]I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of t

that fixed it for me too.

[rocmmixer]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Not glamorous advice, but repeatable baselines save a lot of time.

[batchsize]

Appreciate the detail. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.