what music do you benchmark to

[deltastation]

Did a bit of testing and thought I'd post the result here.

After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.

Happy to post more details if anyone wants them.

[rasterhall]

That matches what I was seeing. The pattern around KV cache pressure has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[signaltrack44]

Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cozygrid12]

I hit almost the same issue last night.

[vectorpoint]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[kernelmike]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[cobalttinker]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Once you know the plain path works, then add the faster path back in piece by piece.

[sparrowgrid]

I hit almost the same issue last night.

[edgeriver]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[paperport64]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.