+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG recall vs speed tradeoff (/showthread.php?tid=595)
RAG recall vs speed tradeoff - cacheline - 09-30-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of MLX and A6000 felt surprisingly usable once I tuned around batch size.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: RAG recall vs speed tradeoff - audiodesk - 09-30-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
I'm happy to test one more variant if you want a second data point.