+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: retrieval stack for private docs (/showthread.php?tid=119)
retrieval stack for private docs - emberquill - 09-03-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like reranking decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
Re: retrieval stack for private docs - macrolab - 09-04-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.
Re: retrieval stack for private docs - vramghost - 09-05-2026
Quote:I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up
Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: retrieval stack for private docs - bytestack - 09-05-2026
Interesting, I hadn't tried that. That matches what I saw on M3 Max.
Re: retrieval stack for private docs - visionforge - 09-05-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: retrieval stack for private docs - embershed99 - 09-08-2026
Appreciate the detail. I ended up in about the same place using LM Studio. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: retrieval stack for private docs - edgequill - 09-08-2026
A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.