quant support gaps for Mixtral

[syshouse]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate router policy until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

[batchcraft]

This keeps coming up in conversation so I figured I'd start a thread.

Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[pixelcrate82]

following this because I'm in the same boat.

[northcraft90]

Appreciate the detail. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routecrate]

Appreciate the detail. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[kernelmike]

[quote]Appreciate the detail. The pattern around token cost has been pretty similar for me. Worth rechecking your defaults though, because one bad ass

Did a small comparison this afternoon. On Threadripper box with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.

[routecrate]

following this because I'm in the same boat.

[kernelmike]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.

[torchtrack24]

Good point. I ended up in about the same place using llama.cpp.

[rasterbench]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.