Multimodal discussion

[southtinker]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[mintstream9]

Spent some time on this today and wanted to compare notes.

same here on my side.

[routearch]

Spent some time on this today and wanted to compare notes.

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[northgrid33]

following this because I'm in the same boat.

[vramscope]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

[cozyarch]

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[visionforge]

same here on my side.

[orbitgarden]

Ran a quick check on my box. On GB200 with llama.cpp, I saw basically the same behavior.

[lablab]

Ran a quick check on my box. On GB200 with llama.cpp, I saw basically the same behavior.

Thanks, that helps. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. Turns out 'just one more test' was a lie.

[dockbrook21]

Good point. The pattern around KV cache pressure has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.