TensorRT-LLM worth the setup time with SGLang

[torchshed66]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

[macrotinker]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[northcraft90]

Need a little more detail here. What model, what runtime, and roughly what context size?

[northgrid33]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[routecrate]

I think the bottleneck is somewhere else. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[macrotinker]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

If you post your exact settings, people here can probably narrow it down pretty fast.

[kernelgrid]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.