Blackwell deployment chatter

[latentgarden]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate batch size until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.

[llm_wren]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[edgecoast]

that fixed it for me too.

I'm happy to test one more variant if you want a second data point.