+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Distillation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=55)
+--- Thread: RAG chunking that actually helped for Qwen (/showthread.php?tid=833)
RAG chunking that actually helped for Qwen - sparrowgrid - 10-07-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Gemma using TensorRT-LLM on 7900 XTX. The headline result was better than I expected, but small local stacks still ended up being the limiting factor.
Curious if anyone else landed in the same place.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: RAG chunking that actually helped for Qwen - cacheline - 10-07-2026
My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: RAG chunking that actually helped for Qwen - bytesupply76 - 10-08-2026
Not sure I agree there. I've seen people blame quant drift when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.