rackmount vs tower for homelab inference

[latentpeak22]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and Ollama for Llama. The interesting difference for me has been HBM constraints.

What would you pick if you had to live with it for a few months?

[rocmcrate26]

A couple questions before I guess too hard. Are you testing this on GB200 or something else?

[cozygrid12]

Ran a quick check on my box. On M3 Max with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[tensorport]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[homelabhenry]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. The logs were somehow calmer than I was.

[tensorport]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[vramghost]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.

[rocmriley]

That matches what I was seeing. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.