Best workstation layout for local models

[latentpeak22]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and llama.cpp for FLUX. The interesting difference for me has been consumer GPUs.

Trying to avoid rebuilding the whole thing twice.

[rocmriley]

Spent some time on this today and wanted to compare notes.

same here on my side. The logs were somehow calmer than I was.

[sparrowworks88]

Spent some time on this today and wanted to compare notes.

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[torchkite]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[vramstream]

worth checking the driver first.

[tensorworks]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[vramghost]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, no

I'd start with the simpler stack.

[kernelhouse70]

Tried to reproduce it just now. On 7900 XTX with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[vectorcrate98]

I don't think that's the whole story. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

[tensorport]

I don't think that's the whole story. I get why that seems right, but on mini PC the bottleneck felt elsewhere.

I tested this tonight. On GB200 with LM Studio, I saw basically the same behavior.