what changed after the inference shift with Ollama

[kernelmike]

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

[prompttinker44]

Good point. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorcrate98]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[stackarch]

following this because I'm in the same boat.

[rocmpath]

following this because I'm in the same boat.

following this because I'm in the same boat.

[turboscope5]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[macronest]

I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[rocmpath]

A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[signalcraft14]

What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmpath]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable. At this point the real bottleneck is my patience.