RAG chunking that actually helped with TensorRT-LLM

[relaysupply90]

I'm trying to sanity-check my setup.

My current setup is Ollama with B200, and I keep running into batch size.

Would appreciate practical advice rather than theory.

[mangotinker73]

Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[northgarden37]

Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[mangowire]

Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[labline]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[thetarunner]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[oxidestream]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem su

Thanks, that helps. The pattern around long context has been pretty similar for me.

[rocmriley]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[routecraft]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[llmarch51]

same here on my side.