+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: recipes for a good local stack with TensorRT-LLM (/showthread.php?tid=305)
recipes for a good local stack with TensorRT-LLM - mlxgarden - 09-16-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.
I'm especially interested in mistakes worth avoiding.
Re: recipes for a good local stack with TensorRT-LLM - rasterbench - 09-18-2026
same here on my side.
Re: recipes for a good local stack with TensorRT-LLM - rocmriley - 09-18-2026
A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: recipes for a good local stack with TensorRT-LLM - thetaember38 - 09-18-2026
Quote:A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.
Re: recipes for a good local stack with TensorRT-LLM - edgeriver - 09-19-2026
subscribed, curious where this lands.
Re: recipes for a good local stack with TensorRT-LLM - cacheline - 09-19-2026
Quote:subscribed, curious where this lands.
Appreciate the detail. The pattern around VRAM headroom has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.