+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Benchmarks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=65)
+--- Thread: eval setup for local models with TensorRT-LLM (/showthread.php?tid=387)
eval setup for local models with TensorRT-LLM - sparrowworks88 - 09-20-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between Ollama and MLX for DeepSeek. The interesting difference for me has been long context.
Interested in tradeoffs more than fan loyalty.
Re: eval setup for local models with TensorRT-LLM - batchpeak - 09-20-2026
Appreciate the detail. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: eval setup for local models with TensorRT-LLM - deltastation - 09-20-2026
A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: eval setup for local models with TensorRT-LLM - northgrid33 - 09-23-2026
I'd start with the simpler stack.
Re: eval setup for local models with TensorRT-LLM - cacheline - 09-23-2026
Quote:I'd start with the simpler stack.
Interesting, I hadn't tried that. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: eval setup for local models with TensorRT-LLM - kernelarch - 09-23-2026
Thanks, that helps. The pattern around reranking has been pretty similar for me. Turns out 'just one more test' was a lie.
Re: eval setup for local models with TensorRT-LLM - phaseport36 - 09-24-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Once you know the plain path works, then add the faster path back in piece by piece.
Re: eval setup for local models with TensorRT-LLM - localtinker - 09-25-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.