tokens/sec tuning ideas

[orbitgarden]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with VRAM headroom.

Curious whether that lines up with what other people are seeing. Turns out 'just one more test' was a lie.

[byteyard]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[promptfield]

Need a little more detail here. Is this local-only or are you routing anything externally?

[kernelghost]

Need a little more detail here. Is this local-only or are you routing anything externally?

Need a little more detail here. What model, what runtime, and roughly what context size?

[vrambandit]

Thanks, that helps. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[macrostack69]

Appreciate the detail. The pattern around quant drift has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vectorfield]

I'd start with the simpler stack.

[oxidegrove40]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[vrambandit]

Need a little more detail here. Are you testing this on Threadripper box or something else?

[bytelab]

Need a little more detail here. Are you testing this on Threadripper box or something else?

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.