+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: JAX for inference only: what are you seeing? (/showthread.php?tid=429)
Pages:
1
2
JAX for inference only: what are you seeing? - torchstation - 09-22-2026
I'm trying to sanity-check my setup.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tool calling.
Would appreciate practical advice rather than theory.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: JAX for inference only: what are you seeing? - localpatch78 - 09-23-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: JAX for inference only: what are you seeing? - vectorshed43 - 09-23-2026
I hit almost the same issue last night.
Re: JAX for inference only: what are you seeing? - stackpilot - 09-24-2026
Quote:I hit almost the same issue last night.
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: JAX for inference only: what are you seeing? - batchpeak - 09-24-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.
Re: JAX for inference only: what are you seeing? - vectorport - 09-24-2026
Maybe, but I had the opposite result. I've seen people blame embeddings when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: JAX for inference only: what are you seeing? - quantghost - 09-24-2026
A couple things I'd check first. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: JAX for inference only: what are you seeing? - routeforge64 - 09-24-2026
Tried to reproduce it just now. On GB200 with vLLM, I saw basically the same behavior.
Re: JAX for inference only: what are you seeing? - orbitbench6 - 09-24-2026
Tried to reproduce it just now. On 4090 with Ollama, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: JAX for inference only: what are you seeing? - sanescope - 09-24-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.