when a plain Python stack is enough

[dockriver]

that fixed it for me too.

[orbitgarden]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[vectorfield]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one. Turns out 'just one more test' was a lie.

[dockember]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there.

[amberpatch25]

Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.