multimodal local stack that feels real on M3 Max

[paperrunner]

Wanted to compare notes with people doing similar things.

My current setup is MLX with Threadripper box, and I keep running into MoE serving.

Would appreciate practical advice rather than theory.

[cachelab]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[audiopath66]

Interesting, I hadn't tried that. The pattern around structured output has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[forgemixer]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Once you know the plain path works, then add the faster path back in piece by piece.

[audiopath66]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Once you know the plain path works, then add the faster path back in piece by piece.

[turboarch95]

Appreciate the detail. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sparrowworks88]

following this because I'm in the same boat.

[quietarch]

That matches what I was seeing. That matches what I saw on GB200.

[latentpeak22]

Tried to reproduce it just now. On 7900 XTX with LM Studio, I saw basically the same behavior.

[visionforge]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.