+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG chunking that actually helped: worth it? (/showthread.php?tid=544)
Pages:
1
2
RAG chunking that actually helped: worth it? - dockbrook21 - 09-28-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about long context.
Simple and boring is a valid answer.
This is mostly for local work rather than public serving.
Re: RAG chunking that actually helped: worth it? - fastfield4 - 09-28-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time. Turns out 'just one more test' was a lie.
Re: RAG chunking that actually helped: worth it? - localpatch - 09-28-2026
Quote:A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous
I think the bottleneck is somewhere else. I've seen people blame workstation thermals when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: RAG chunking that actually helped: worth it? - audioforge - 09-28-2026
Appreciate the detail. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: RAG chunking that actually helped: worth it? - quietcraft - 09-28-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: RAG chunking that actually helped: worth it? - bytesupply76 - 09-28-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.
Re: RAG chunking that actually helped: worth it? - mintstation99 - 09-28-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamo
That matches what I was seeing. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. I keep promising myself this is the last rebuild.
Re: RAG chunking that actually helped: worth it? - localpatch - 09-28-2026
Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: RAG chunking that actually helped: worth it? - llmtrack - 09-30-2026
Need a little more detail here. Are you testing this on 7900 XTX or something else?
Re: RAG chunking that actually helped: worth it? - macrostack69 - 09-30-2026
What exact setup are you running? Are you testing this on GB200 or something else?