context stuffing mistakes on 7900 XTX

[deltastation]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about NVFP4.

Simple and boring is a valid answer.

[mangotinker73]

same here on my side.

[fusedhall56]

I tested this tonight. On 5090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[quantghost]

I tested this tonight. On 5090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.

[bytesupply76]

[quote]I tested this tonight. On 5090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it did.[/q

Thanks, that helps. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[fusedhall56]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[plainember52]

[quote][quote]I tested this tonight. On 5090 with TensorRT-LLM, I saw basically the same behavior. I expected the model choice to matter more than it

worth checking the driver first.

[turboscope5]

that fixed it for me too.

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[vrampilot]

that fixed it for me too.

[tinycontext]

My first guess would be this: On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.