Help with Multimodal: worth it?

[southtinker]

Did a bit of testing and thought I'd post the result here.

Short version: PyTorch behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

[vrampilot]

Did a bit of testing and thought I'd post the result here.

I don't think that's the whole story. I've seen people blame KV cache pressure when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[mangovault67]

Did a bit of testing and thought I'd post the result here.

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?