RAG chunking that actually helped for Llama

[dockkite69]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and llama.cpp for Qwen. The interesting difference for me has been speculative decoding.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.