+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: retrieval stack for private docs (/showthread.php?tid=790)
Pages:
1
2
retrieval stack for private docs - cacheline - 10-06-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.
Simple and boring is a valid answer.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: retrieval stack for private docs - vramarch - 10-06-2026
Thanks, that helps. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: retrieval stack for private docs - deepseeked - 10-08-2026
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: retrieval stack for private docs - sparrowbrook - 10-08-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.
Re: retrieval stack for private docs - circuitworks90 - 10-08-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once yo
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: retrieval stack for private docs - routecraft - 10-08-2026
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: retrieval stack for private docs - pixelcrate82 - 10-08-2026
Thanks, that helps. That matches what I saw on mini PC. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: retrieval stack for private docs - modbench - 10-08-2026
A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: retrieval stack for private docs - southpatch - 10-09-2026
Quote:A couple things I'd check first. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot
Interesting, I hadn't tried that. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: retrieval stack for private docs - papermixer4 - 10-09-2026
Appreciate the detail. I don't think your conclusion is far off.