image generation model locals actually use

[thetaplane]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

[cozykite]

Appreciate the detail. The pattern around prefix caching has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

If you post your exact settings, people here can probably narrow it down pretty fast.

[southpatch]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[batchsize]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[visionforge]

following this because I'm in the same boat.

[rocmcrate26]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[batchshed]

worth checking the driver first.

[turboyard94]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[vramscope]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[nanopatch]

I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives.