serving on Blackwell

[kernelgrid]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: SGLang on dual 3090, keeping token cost in mind from the start.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[sysstation69]

I hit almost the same issue last night.

[torchtrack24]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rasterbench]

Interesting, I hadn't tried that. The pattern around HBM constraints has been pretty similar for me.

[macrotinker]

Appreciate the detail. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.

[syshouse]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[macrolab]

I'd start with the simpler stack.

[papermixer4]

A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious.

[batchcraft]

Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[kernelmike]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.