Best workstation layout for local models: anyone testing this?

[edgequill]

that fixed it for me too.

[tensorworks]

That matches what I was seeing. The pattern around continuous batching has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[homelabhenry]

Not sure I agree there. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one. The logs were somehow calmer than I was.

[batchwire]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box.

[cozygrid12]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box.

Ran a quick check on my box. On B200 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.