PyTorch compile got faster? on GB200

[deltastation]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on A6000, keeping HBM constraints in mind from the start.

Feel free to tear it apart if you see something dumb.

[visionforge]

Thanks, that helps. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetaplane]

Thanks, that helps. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more op

subscribed, curious where this lands.

[benchplease]

I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[southspark]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[torchkite]

Maybe, but I had the opposite result. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[thetarunner]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

I'm happy to test one more variant if you want a second data point.

[kernelmike]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know the plain path works, then add the faster path back in piece by piece.

[torchshed]

worth checking the driver first.

[deltastation]

Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.