working config for 32k context for Mistral

[vectorport]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.

What would you pick if you had to live with it for a few months?

[audioriver]

Spent some time on this today and wanted to compare notes.

Tried to reproduce it just now. On GB200 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.