PCIe lane weirdness with inference loads for Mixtral

[embersmith]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tokens/sec.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[signaltrack44]

Good point. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cudapoint]

Thanks, that helps. The pattern around VRAM headroom has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[griffinloom66]

I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[turbogrid11]

My first guess would be this: If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[turboarch95]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[rocmriley]

I hit almost the same issue last night.

[batchwire]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[amberhall]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Not glamorous advice, but repeatable baselines save a lot of time.

[sanevault]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. No

I hit almost the same issue last night.