Multimodal discussion

[visionforge]

Did a bit of testing and thought I'd post the result here.

I tried this with Kimi using MLX on 4090. The headline result was better than I expected, but KV cache pressure still ended up being the limiting factor.

Curious if anyone else landed in the same place.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[southtinker]

Did a bit of testing and thought I'd post the result here.

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[macropilot]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[kernelmike]

worth checking the driver first.

[quietcraft]

worth checking the driver first.

subscribed, curious where this lands.

[vramscope]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[mintstream9]

What exact setup are you running? Is this local-only or are you routing anything externally? Turns out 'just one more test' was a lie.

[batchwire]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[sparrowgrid]

Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[torchshed66]

I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.