+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Video (https://www.datasculpting.ai/inference/forumdisplay.php?fid=70)
+--- Thread: multimodal local stack that feels real (/showthread.php?tid=1061)
Pages:
1
2
multimodal local stack that feels real - mangohouse - 10-12-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: multimodal local stack that feels real - visionforge - 10-12-2026
That matches what I was seeing. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: multimodal local stack that feels real - oxideworks59 - 10-13-2026
I'd start with the simpler stack.
Re: multimodal local stack that feels real - cachelab - 10-13-2026
that fixed it for me too.
Re: multimodal local stack that feels real - mangohouse - 10-16-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at continuous batching. Once you know the plain path works, then add the faster path back in piece by piece.
Re: multimodal local stack that feels real - turboarch95 - 10-17-2026
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: multimodal local stack that feels real - latentpeak22 - 10-17-2026
What worked for me was pretty simple. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: multimodal local stack that feels real - quiettoken - 10-18-2026
following this because I'm in the same boat.
Re: multimodal local stack that feels real - mangohouse - 10-18-2026
Quote:following this because I'm in the same boat.
Good point. I ended up in about the same place using LM Studio. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: multimodal local stack that feels real - thetaember38 - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals.