non-AI thing you built recently with TensorRT-LLM

[dockkite69]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Curious whether that lines up with what other people are seeing.

[bytepilot]

following this because I'm in the same boat.

[sanevault]

same here on my side.

[dockworks23]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[phasespark]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[fastlab]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[dockworks23]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[torchsmith25]

that fixed it for me too.