tokens/sec tuning ideas: worth it?

[macronest]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, SGLang, A6000, with a lot of attention on consumer GPUs.

Feel free to tear it apart if you see something dumb.

[northhall88]

Did a small comparison this afternoon. On mini PC with vLLM, I saw basically the same behavior.

[marbletrack]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

If you post your exact settings, people here can probably narrow it down pretty fast.

[cozygrid12]

Good point. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. The logs were somehow calmer than I was.

[vectorhouse71]

I hit almost the same issue last night.

[northgarden37]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[northgrid33]

What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[cachelab]

That matches what I was seeing. That matches what I saw on GB200.

[macrotinker]

That matches what I was seeing. That matches what I saw on GB200.

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[sparrowghost]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.