rackmount vs tower for homelab inference on dual 3090

[batchyard]

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling batch size.

If you've tested something similar, what did you keep and what did you stop bothering with?

[tensorport]

Can you fill in a bit more context? Are you testing this on B200 or something else?

[vramghost]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[phasenode]

following this because I'm in the same boat.

[stackpilot]

I don't think that's the whole story. I've seen people blame reranking when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[tensorport]

Interesting, I hadn't tried that. That matches what I saw on Threadripper box.

[batchwire]

A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cobaltlab]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.

[homelabhenry]

I hit almost the same issue last night.

[tensorworks]

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.