VRAM bottleneck on consumer cards on M3 Max

[sparrowhall]

Maybe I'm missing something obvious.

My current setup is PyTorch with Threadripper box, and I keep running into VRAM headroom.

Would appreciate practical advice rather than theory.

[thetarunner]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[torchshed]

Need a little more detail here. Is this local-only or are you routing anything externally?