disaggregated inference thoughts on dual 3090

[prompttinker44]

Did a bit of testing and thought I'd post the result here.

Short version: LM Studio behaved well for me, but only after I stopped ignoring structured output.

Nothing scientific here, just practical notes from a real setup.

[stackarch]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[rocmpath]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[kernelmike]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[tensorfield88]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[sparrowghost]

Thanks, that helps. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

If you post your exact settings, people here can probably narrow it down pretty fast.

[vramport]

that fixed it for me too.

[plainline]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[kernelmike]

Good point. The pattern around small local stacks has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[signalcraft14]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.