best local open-weight model this week

[tensorport]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tail latency decides everything downstream.

Curious whether that lines up with what other people are seeing.

[benchplease]

What exact setup are you running? Is this local-only or are you routing anything externally?

[sanescope]

What exact setup are you running? Is this local-only or are you routing anything externally?

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorfield88]

Ran a quick check on my box. On 7900 XTX with PyTorch, I saw basically the same behavior.

[griffinloom66]

I'd start with the simpler stack.

[oxidestream]

I hit almost the same issue last night.

[emberquill]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency.

[oxidemixer]

I'd start with the simpler stack.

[tensorfield88]

Appreciate the detail. I don't think your conclusion is far off.

[plainline]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.