+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Video (https://www.datasculpting.ai/inference/forumdisplay.php?fid=70)
+--- Thread: multimodal local stack that feels real on GB200 (/showthread.php?tid=789)
Pages:
1
2
multimodal local stack that feels real on GB200 - batchwire - 10-05-2026
Did a bit of testing and thought I'd post the result here.
Short version: MLX behaved well for me, but only after I stopped ignoring reranking.
Happy to post more details if anyone wants them. At this point the real bottleneck is my patience.
Re: multimodal local stack that feels real on GB200 - cobalttinker - 10-05-2026
Quote:Did a bit of testing and thought I'd post the result here.
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: multimodal local stack that feels real on GB200 - audiopath66 - 10-06-2026
Did a small comparison this afternoon. On M3 Max with LM Studio, I saw basically the same behavior. The main thing that moved was latency, not quality.
Re: multimodal local stack that feels real on GB200 - paperrunner - 10-06-2026
Quote:Did a small comparison this afternoon. On M3 Max with LM Studio, I saw basically the same behavior. The main thing that moved was latency, not quality
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: multimodal local stack that feels real on GB200 - visionforge - 10-06-2026
I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. I keep promising myself this is the last rebuild.
Re: multimodal local stack that feels real on GB200 - cobalttinker - 10-07-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: multimodal local stack that feels real on GB200 - quietarch - 10-07-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Re: multimodal local stack that feels real on GB200 - cobaltworks25 - 10-07-2026
Quote:I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: multimodal local stack that feels real on GB200 - cachelab - 10-09-2026
I hit almost the same issue last night.
Re: multimodal local stack that feels real on GB200 - thetaember38 - 10-09-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency.