+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Optimization & Performance (https://www.datasculpting.ai/inference/forumdisplay.php?fid=10)
+--- Forum: Performance Tuning (https://www.datasculpting.ai/inference/forumdisplay.php?fid=42)
+--- Thread: what changed after the inference shift with Ollama (/showthread.php?tid=1192)
Pages:
1
2
what changed after the inference shift with Ollama - kernelmike - 10-13-2026
Did a bit of testing and thought I'd post the result here.
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring quant drift.
Nothing scientific here, just practical notes from a real setup.
This is mostly for local work rather than public serving.
Re: what changed after the inference shift with Ollama - prompttinker44 - 10-14-2026
Good point. The pattern around VRAM headroom has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: what changed after the inference shift with Ollama - vectorcrate98 - 10-15-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.
Re: what changed after the inference shift with Ollama - stackarch - 10-18-2026
following this because I'm in the same boat.
Re: what changed after the inference shift with Ollama - rocmpath - 10-18-2026
Quote:following this because I'm in the same boat.
following this because I'm in the same boat.
Re: what changed after the inference shift with Ollama - turboscope5 - 10-18-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: what changed after the inference shift with Ollama - macronest - 10-18-2026
I don't think that's the whole story. I've seen people blame structured output when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: what changed after the inference shift with Ollama - rocmpath - 10-18-2026
A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: what changed after the inference shift with Ollama - signalcraft14 - 10-18-2026
What worked for me was pretty simple. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: what changed after the inference shift with Ollama - rocmpath - 10-18-2026
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable. At this point the real bottleneck is my patience.