multimodal local stack that feels real: what are you seeing?

[visionforge]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.

I'm especially interested in mistakes worth avoiding.

[visionforge]

That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[phasespark]

Did a small comparison this afternoon. On GB200 with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.

[visionforge]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[cachelab]

that fixed it for me too.

[visionforge]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[latentpeak22]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

worth checking the driver first.

[visionforge]

following this because I'm in the same boat.

[cobalttinker]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[paperrunner]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.