Multimodal discussion

[visionforge]

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around quant drift.

Happy to post more details if anyone wants them.

I care more about stability than squeezing the last possible tokens/sec out of it.

[lablab]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[vramarch]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[macrostack69]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[macropilot]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[fastquill74]

Appreciate the detail. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[macrostack69]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. Once you know the plain path works, then add the faster path back in piece by piece.

[torchshed66]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. Once you

I don't think that's the whole story. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[macropilot]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box.

[mangovault67]

Appreciate the detail. The pattern around reranking has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.