best local open-weight model this week

[circuitwire58]

Did a bit of testing and thought I'd post the result here.

Short version: LM Studio behaved well for me, but only after I stopped ignoring speculative decoding.

Nothing scientific here, just practical notes from a real setup.

[sparrowghost]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. Once you know the plain path works, then add the faster path back in piece by piece.

[bytestack]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[amberplane]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals.

[sparrowshed]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[amberkite]

Thanks, that helps. That matches what I saw on 7900 XTX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mellowstream]

Tried to reproduce it just now. On A6000 with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[bytestack]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[thetaember38]

worth checking the driver first.

[mellowstream]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.