+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: JAX for inference only (/showthread.php?tid=1439)
Pages:
1
2
Re: JAX for inference only - audiodesk - 10-18-2026
Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
I'm happy to test one more variant if you want a second data point.
Re: JAX for inference only - routecrate - 10-18-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.
Re: JAX for inference only - macronest - 10-19-2026
Quote:Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output. Once you know the plain path works, then add the faster path back in piece by piece.
Re: JAX for inference only - tensorworks - 10-19-2026
Quote:I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once
Ran a quick check on my box. On A6000 with SGLang, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: JAX for inference only - dockridge - 10-19-2026
worth checking the driver first.
Re: JAX for inference only - kernelghost - 10-19-2026
Quote:Quote:Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal.
worth checking the driver first.