framework overhead in real deployments

[kernelghost]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[fusedhall]

following this because I'm in the same boat.

[llmarch83]

Thanks, that helps. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[sanescope]

following this because I'm in the same boat.

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.