serving on Blackwell for Phi

[pixelcrate82]

I figured I'd ask here before changing everything.

I'm mostly working with Phi in TensorRT-LLM on Threadripper box. The part I'm trying to understand is quant drift.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[macrotinker]

What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[sysstation69]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[northcraft90]

Need a little more detail here. Is this local-only or are you routing anything externally?

[kernelpeak]

Need a little more detail here. Is this local-only or are you routing anything externally?

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[torchtrack24]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[kernelgrid]

Thanks, that helps. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[torchtrack24]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[routecrate]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.

[rocmmixer]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.