10-07-2026, 12:31 AM
Spent some time on this today and wanted to compare notes.
I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been batch size.
What would you pick if you had to live with it for a few months?
The part that surprised me was how much the boring details mattered more than the model choice.
I've gone back and forth between TensorRT-LLM and llama.cpp for Mistral. The interesting difference for me has been batch size.
What would you pick if you had to live with it for a few months?
The part that surprised me was how much the boring details mattered more than the model choice.
Apple Silicon • MLX mostly