RAG chunking that actually helped for Llama

[dockkite69]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and llama.cpp for Qwen. The interesting difference for me has been speculative decoding.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.

[cudascope]

Good point. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vramghost]

Appreciate the detail. That matches what I saw on B200. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.