new here, what's your stack with SGLang

[tensorkite]

Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me.

[rocmrunner40]

Interesting, I hadn't tried that. The pattern around INT4 has been pretty similar for me.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. Once you know the plain path works, then add the faster path back in piece by piece.

[vramghost]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmplane]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[cacheplane98]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.