+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context for Mistral (/showthread.php?tid=1002)
working config for 32k context for Mistral - vectorport - 10-10-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.
What would you pick if you had to live with it for a few months?
Re: working config for 32k context for Mistral - audioriver - 10-12-2026
Quote:Spent some time on this today and wanted to compare notes.
Tried to reproduce it just now. On GB200 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.