+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG chunking that actually helped on Threadripper box (/showthread.php?tid=999)
Pages:
1
2
RAG chunking that actually helped on Threadripper box - pixelspark - 10-10-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Phi, Ollama, Threadripper box, with a lot of attention on quant drift.
Feel free to tear it apart if you see something dumb.
Re: RAG chunking that actually helped on Threadripper box - kernelmike - 10-12-2026
Appreciate the detail. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: RAG chunking that actually helped on Threadripper box - plainline - 10-12-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: RAG chunking that actually helped on Threadripper box - torchdesk - 10-12-2026
I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: RAG chunking that actually helped on Threadripper box - latentgarden - 10-12-2026
I'd start with the simpler stack.
Re: RAG chunking that actually helped on Threadripper box - vectorpoint - 10-12-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
Re: RAG chunking that actually helped on Threadripper box - visionyard23 - 10-12-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: RAG chunking that actually helped on Threadripper box - rocmpath - 10-12-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: RAG chunking that actually helped on Threadripper box - localcraft17 - 10-13-2026
subscribed, curious where this lands.
Re: RAG chunking that actually helped on Threadripper box - thetanest - 10-13-2026
following this because I'm in the same boat.