prefill bottlenecks on longer prompts

[bytegarden73]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: TensorRT-LLM on B200, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

This is mostly for local work rather than public serving.

[vrambandit]

Thought I'd post a small recipe because a lot of threads ask similar things.

I hit almost the same issue last night.