RAG chunking that actually helped with TensorRT-LLM

RAG chunking that actually helped with TensorRT-LLM - Printable Version

+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG chunking that actually helped with TensorRT-LLM (/showthread.php?tid=1054)

Pages: 1 2

Re: RAG chunking that actually helped with TensorRT-LLM - thetagrid15 - 10-16-2026

same here on my side.

Re: RAG chunking that actually helped with TensorRT-LLM - rocmcrate26 - 10-16-2026

Quote:same here on my side.

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

Re: RAG chunking that actually helped with TensorRT-LLM - localtinker - 10-17-2026

That matches what I was seeing. I ended up in about the same place using vLLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.