CUDA update broke my throughput

[cobaltspark]

Curious what people here are seeing.

What exact setup are you running? Is this local-only or are you routing anything externally?

[mintforge]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs.

[dockember]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[nanobrook]

Interesting, I hadn't tried that. That matches what I saw on M3 Max.

Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.