quant support gaps

[sysstation69]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, LM Studio, A6000, with a lot of attention on INT4.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[routeforge64]

That matches what I was seeing. I ended up in about the same place using llama.cpp.

[kernelgrid]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[batchcraft]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[routecrate]

Interesting, I hadn't tried that. I don't think your conclusion is far off.

[torchtrack24]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[rocmriley]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

Thanks, that helps. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[batchscope72]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[sysstation69]

following this because I'm in the same boat.

[ggufvault]

following this because I'm in the same boat.