JAX for inference only on dual 3090

[quiettoken]

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[latentpeak22]

Appreciate the detail. I don't think your conclusion is far off.

[deltapoint17]

Not sure I agree there. I've seen people blame token cost when the real issue was much more mundane.

[torchshed]

same here on my side.

[vramghost]

Interesting, I hadn't tried that. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

I'm happy to test one more variant if you want a second data point.