vision-language models on consumer gear with llama.cpp

[benchplease]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around long context.

If someone wants the exact settings I can clean them up and post them.

[thetaember38]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[visionforge]

Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cobaltworks25]

Interesting, I hadn't tried that. That matches what I saw on 7900 XTX. Worth rechecking your defaults though, because one bad assumption can make the

What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.

[phasespark]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[labline]

Interesting, I hadn't tried that. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[visionforge]

I hit almost the same issue last night.

[phaseport36]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Not glamorous advice, but repeatable baselines save a lot of time.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[audiopath66]

What worked for me was pretty simple. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[cachelab]

What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.