NVIDIA driver regression for Phi

[edgecoast]

Did a bit of testing and thought I'd post the result here.

I tried this with Gemma using vLLM on M3 Max. The headline result was better than I expected, but reranking still ended up being the limiting factor.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

[torchshed]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[kernelmike]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[amberpatch25]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[cozysmith]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[edgecoast]

What exact setup are you running? Are you testing this on 4090 or something else?

Good point. The pattern around tail latency has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetapath]

Interesting, I hadn't tried that. That matches what I saw on dual 3090.

[mintforge]

I don't think that's the whole story. I've seen people blame quant drift when the real issue was much more mundane.

[cobaltspark]

What exact setup are you running? Are you testing this on 4090 or something else?

[vectorpoint]

I tested this tonight. On B200 with MLX, I saw basically the same behavior.