RAG chunking that actually helped for Llama - Printable Version +- Inference (https://www.datasculpting.ai/inference) +-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13) +--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55) +--- Thread: RAG chunking that actually helped for Llama (/showthread.php?tid=40)

RAG chunking that actually helped for Llama - dockkite69 - 08-26-2026 Spent some time on this today and wanted to compare notes. I've gone back and forth between TensorRT-LLM and llama.cpp for Qwen. The interesting difference for me has been speculative decoding. What would you pick if you had to live with it for a few months? This is mostly for local work rather than public serving. Re: RAG chunking that actually helped for Llama - cudascope - 08-28-2026 Good point. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. Re: RAG chunking that actually helped for Llama - vramghost - 08-28-2026 Appreciate the detail. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer. There are a few different conversations hiding inside this thread, so being explicit about the goal helps.