+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: PyTorch compile got faster? on GB200 (/showthread.php?tid=473)
Pages:
1
2
PyTorch compile got faster? on GB200 - deltastation - 09-24-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: PyTorch on A6000, keeping HBM constraints in mind from the start.
Feel free to tear it apart if you see something dumb.
Re: PyTorch compile got faster? on GB200 - visionforge - 09-24-2026
Thanks, that helps. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: PyTorch compile got faster? on GB200 - thetaplane - 09-25-2026
Quote:Thanks, that helps. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more op
subscribed, curious where this lands.
Re: PyTorch compile got faster? on GB200 - benchplease - 09-26-2026
I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: PyTorch compile got faster? on GB200 - southspark - 09-26-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
Re: PyTorch compile got faster? on GB200 - torchkite - 09-26-2026
Maybe, but I had the opposite result. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: PyTorch compile got faster? on GB200 - thetarunner - 09-26-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
I'm happy to test one more variant if you want a second data point.
Re: PyTorch compile got faster? on GB200 - kernelmike - 09-28-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know the plain path works, then add the faster path back in piece by piece.
Re: PyTorch compile got faster? on GB200 - torchshed - 09-28-2026
worth checking the driver first.
Re: PyTorch compile got faster? on GB200 - deltastation - 09-28-2026
Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.