tokens/sec tuning ideas

[rocmriley]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like batch size decides everything downstream.

Curious whether that lines up with what other people are seeing.

[oxidemixer]

This keeps coming up in conversation so I figured I'd start a thread.

Interesting, I hadn't tried that. The pattern around router policy has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[thetarunner]

This keeps coming up in conversation so I figured I'd start a thread.

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cozyarch]

that fixed it for me too.

[macroforge]

that fixed it for me too.

Appreciate the detail. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[labnest]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[vramridge]

Good point. That matches what I saw on mini PC.