+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG chunking that actually helped with TensorRT-LLM (/showthread.php?tid=1054)
Pages:
1
2
RAG chunking that actually helped with TensorRT-LLM - relaysupply90 - 10-11-2026
I'm trying to sanity-check my setup.
My current setup is Ollama with B200, and I keep running into batch size.
Would appreciate practical advice rather than theory.
Re: RAG chunking that actually helped with TensorRT-LLM - mangotinker73 - 10-12-2026
Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: RAG chunking that actually helped with TensorRT-LLM - northgarden37 - 10-12-2026
Quote:Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: RAG chunking that actually helped with TensorRT-LLM - mangowire - 10-13-2026
Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
Re: RAG chunking that actually helped with TensorRT-LLM - labline - 10-13-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: RAG chunking that actually helped with TensorRT-LLM - thetarunner - 10-14-2026
What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: RAG chunking that actually helped with TensorRT-LLM - oxidestream - 10-14-2026
Quote:What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem su
Thanks, that helps. The pattern around long context has been pretty similar for me.
Re: RAG chunking that actually helped with TensorRT-LLM - rocmriley - 10-16-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: RAG chunking that actually helped with TensorRT-LLM - routecraft - 10-16-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: RAG chunking that actually helped with TensorRT-LLM - llmarch51 - 10-16-2026
same here on my side.