do small edge boxes make sense: early impressions

[latentscope35]

Thanks, that helps. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[quantsupply]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4.

[routecraft]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4.

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.