70B still worth the pain

[routeline23]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

If you post your exact settings, people here can probably narrow it down pretty fast.

[edgekite]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.

[plaingarden18]

My first guess would be this: On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[deltastation]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.