10-10-2026, 01:47 PM
Spent some time on this today and wanted to compare notes.
I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.
What would you pick if you had to live with it for a few months?
I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.
What would you pick if you had to live with it for a few months?