eval setup for local models with TensorRT-LLM

[sparrowworks88]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and MLX for DeepSeek. The interesting difference for me has been long context.

Interested in tradeoffs more than fan loyalty.

[batchpeak]

Appreciate the detail. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deltastation]

A couple things I'd check first. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[northgrid33]

I'd start with the simpler stack.

[cacheline]

I'd start with the simpler stack.

Interesting, I hadn't tried that. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[kernelarch]

Thanks, that helps. The pattern around reranking has been pretty similar for me. Turns out 'just one more test' was a lie.

[phaseport36]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Once you know the plain path works, then add the faster path back in piece by piece.

[localtinker]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.