serving on Blackwell

[northcraft90]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on Threadripper box, keeping HBM constraints in mind from the start.

If someone wants the exact settings I can clean them up and post them.

This is mostly for local work rather than public serving.

[batchcraft]

following this because I'm in the same boat.

[torchtrack24]

I'd start with the simpler stack.

[batchcraft]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[routecrate]

That matches what I was seeing. I don't think your conclusion is far off.

[tensorriver]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[northcraft90]

What exact setup are you running? Is this local-only or are you routing anything externally?

[kernelmike]

following this because I'm in the same boat.

[quiettoken]

Thanks, that helps. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelmike]

Thanks, that helps. That matches what I saw on A6000. The biggest change for me was simplifying the stack instead of adding one more optimization laye

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?