+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: JAX for inference only on dual 3090 (/showthread.php?tid=1121)
Pages:
1
2
JAX for inference only on dual 3090 - homelabhenry - 10-12-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and 4090.
I'm especially interested in mistakes worth avoiding.
This is mostly for local work rather than public serving.
Re: JAX for inference only on dual 3090 - audioforge - 10-12-2026
subscribed, curious where this lands.
I'm happy to test one more variant if you want a second data point.
Re: JAX for inference only on dual 3090 - sparrowworks88 - 10-12-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Re: JAX for inference only on dual 3090 - tinycontext - 10-12-2026
Quote:My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: JAX for inference only on dual 3090 - pixelcrate82 - 10-12-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: JAX for inference only on dual 3090 - deepseeked - 10-12-2026
A couple things I'd check first. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: JAX for inference only on dual 3090 - dockember - 10-13-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Re: JAX for inference only on dual 3090 - vrambandit - 10-13-2026
Thanks, that helps. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer. The logs were somehow calmer than I was.
Re: JAX for inference only on dual 3090 - quietarch - 10-13-2026
Appreciate the detail. The pattern around HBM constraints has been pretty similar for me.
Re: JAX for inference only on dual 3090 - thetashed24 - 10-14-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.