Apple unified memory limits

[audiodesk]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like quant drift decides everything downstream.

Wondering if this is just where the whole space is heading now.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[orbitline]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[quietarch]

Good point. I ended up in about the same place using TensorRT-LLM.