+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Video (https://www.datasculpting.ai/inference/forumdisplay.php?fid=70)
+--- Thread: multimodal local stack that feels real for Qwen (/showthread.php?tid=638)
Pages:
1
2
multimodal local stack that feels real for Qwen - phasespark - 10-01-2026
Did a bit of testing and thought I'd post the result here.
Short version: llama.cpp behaved well for me, but only after I stopped ignoring tokens/sec.
Nothing scientific here, just practical notes from a real setup.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: multimodal local stack that feels real for Qwen - forgemixer - 10-01-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: multimodal local stack that feels real for Qwen - visionforge - 10-01-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: multimodal local stack that feels real for Qwen - oxideworks59 - 10-01-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: multimodal local stack that feels real for Qwen - turboarch95 - 10-01-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: multimodal local stack that feels real for Qwen - visionforge - 10-01-2026
Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: multimodal local stack that feels real for Qwen - cobalttinker - 10-03-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: multimodal local stack that feels real for Qwen - batchsize - 10-03-2026
Quote:I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before th
What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: multimodal local stack that feels real for Qwen - paperrunner - 10-03-2026
Interesting, I hadn't tried that. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: multimodal local stack that feels real for Qwen - cobaltworks25 - 10-06-2026
Ran a quick check on my box. On mini PC with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.