quant choices for 70B local: what are you seeing?

[vectorport]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and vLLM for DeepSeek. The interesting difference for me has been VRAM headroom.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

[relaynest82]

I hit almost the same issue last night.

[llmarch83]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

I'm happy to test one more variant if you want a second data point.

[southtinker]

same here on my side.

[northquill]

What exact setup are you running? Are you testing this on Threadripper box or something else? The logs were somehow calmer than I was.

[benchplease]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling.

[bytehall]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmarch51]

Ran a quick check on my box. On Threadripper box with LM Studio, I saw basically the same behavior.

[vrambandit]

following this because I'm in the same boat.

[fusedtinker]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

I'm happy to test one more variant if you want a second data point.