CUDA update broke my throughput with Ollama

[kernelmike]

Did a bit of testing and thought I'd post the result here.

I tried this with DeepSeek using Ollama on 4090. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

[rasterhall]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[northgarden37]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[dockember]

worth checking the driver first.

[sparrowkite53]

My first guess would be this: If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious.

[edgecoast]

Interesting, I hadn't tried that. The pattern around consumer GPUs has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetapath]

I hit almost the same issue last night.

[kernelmike]

Not sure I agree there. I get why that seems right, but on M3 Max the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[audiopath66]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[promptspark58]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time. I keep promising myself this is the last rebuild.