benchmark result I don't trust

[mellowwire98]

Did a bit of testing and thought I'd post the result here.

I tried this with Llama using PyTorch on M3 Max. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

[sparrowworks88]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I keep promising myself this is the last rebuild.

[quiethouse]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[vectorfield]

Good point. The pattern around NVFP4 has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[localtinker]

worth checking the driver first.

[llmarch83]

following this because I'm in the same boat.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[dockridge]

worth checking the driver first.