multimodal local stack that feels real: what are you seeing?

[latentpeak22]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

[thetaember38]

Tried to reproduce it just now. On A6000 with llama.cpp, I saw basically the same behavior. The logs were somehow calmer than I was.

[quietarch]

Thanks, that helps. The pattern around tokens/sec has been pretty similar for me.

[visionforge]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[southtinker]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plai

Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[visionforge]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Not glamorous advice, but repeatable baselines save a lot of time.