quant choices for 70B local for Llama

[llmarch51]

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling VRAM headroom.

If you've tested something similar, what did you keep and what did you stop bothering with?

[vrambandit]

Good point. The pattern around speculative decoding has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[southtinker]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[northquill]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the

My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vrambandit]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[quiettoken]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at NVFP4. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.