framework overhead in real deployments: early impressions

[llm_wren]

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring structured output.

Curious if anyone else landed in the same place.

[rocmrunner40]

A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[latentpeak22]

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[forgemixer]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[torchsupply99]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[kernelmike]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure show

same here on my side.

[griffinloom66]

[quote]Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressu

following this because I'm in the same boat.