rackmount vs tower for homelab inference with llama.cpp

[vectorcrate98]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

[batchyard]

I hit almost the same issue last night.

[tensorworks]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[batchyard]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[cacheline]

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious

worth checking the driver first.

[edgequill]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[northquill]

What exact setup are you running? What model, what runtime, and roughly what context size?

[stackpilot]

Not sure I agree there. I've seen people blame KV cache pressure when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[llmpeak]

Ran a quick check on my box. On mini PC with TensorRT-LLM, I saw basically the same behavior.