NVFP4 / INT4 tradeoffs

[benchplease]

What exact setup are you running? Are you testing this on Threadripper box or something else?

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch.

[visionforge]

Good point. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[marbleyard32]

Good point. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization

Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps. I keep promising myself this is the last rebuild.