local video generation really usable yet for Mistral

[torchdesk]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.

[bytepilot]

following this because I'm in the same boat.

[rasterbench]

That matches what I was seeing. The pattern around workstation thermals has been pretty similar for me.

[vectorpoint]

Good point. That matches what I saw on mini PC.

[tensorkite]

What exact setup are you running? Are you testing this on Threadripper box or something else?

[dockkite69]

following this because I'm in the same boat.

[labline]

following this because I'm in the same boat.

I hit almost the same issue last night.

[nanopatch]

My first guess would be this: On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[modrunner27]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

I'm happy to test one more variant if you want a second data point.

[edgequill]

Good point. The pattern around prefill/decode split has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.