+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: TensorRT-LLM (https://www.datasculpting.ai/inference/forumdisplay.php?fid=37)
+--- Thread: quant support gaps for Mixtral (/showthread.php?tid=1139)
Pages:
1
2
quant support gaps for Mixtral - syshouse - 10-12-2026
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate router policy until they actually try to run this stuff for more than one afternoon.
Wondering if this is just where the whole space is heading now.
Re: quant support gaps for Mixtral - batchcraft - 10-13-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: quant support gaps for Mixtral - pixelcrate82 - 10-14-2026
following this because I'm in the same boat.
Re: quant support gaps for Mixtral - northcraft90 - 10-14-2026
Appreciate the detail. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: quant support gaps for Mixtral - routecrate - 10-15-2026
Quote:Appreciate the detail. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: quant support gaps for Mixtral - kernelmike - 10-15-2026
Quote:[quote]Appreciate the detail. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad ass
Did a small comparison this afternoon. On Threadripper box with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: quant support gaps for Mixtral - routecrate - 10-15-2026
following this because I'm in the same boat.
Re: quant support gaps for Mixtral - kernelmike - 10-15-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.
Re: quant support gaps for Mixtral - torchtrack24 - 10-15-2026
Good point. I ended up in about the same place using llama.cpp.
Re: quant support gaps for Mixtral - rasterbench - 10-16-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.