8B vs 14B sweet spot

[benchplease]

worth checking the driver first.

Maybe, but I had the opposite result. I've seen people blame batch size when the real issue was much more mundane.

[quiettoken]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[emberpeak]

Maybe, but I had the opposite result. I get why that seems right, but on M3 Max the bottleneck felt elsewhere.