best local open-weight model this week with vLLM

[cozykite]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, llama.cpp, mini PC, with a lot of attention on workstation thermals.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

[sysstation69]

same here on my side.

[dockember]

I hit almost the same issue last night.