Blackwell deployment chatter

[phaseport36]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[kernelmike]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[amberpatch25]

that fixed it for me too.

[latentgarden]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[edgecoast]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

If you post your exact settings, people here can probably narrow it down pretty fast.