RAG recall vs speed tradeoff: setup advice

[cudaworks]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[deltastation]

I don't think that's the whole story. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[rocmcrate26]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[routeforge64]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4.

[plainline]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[audiopoint18]

I hit almost the same issue last night.

[quiettoken]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[mangohouse]

Thanks, that helps. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[byteyard]

Thanks, that helps. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the

Good point. The pattern around batch size has been pretty similar for me.

[sysstation69]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.