+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Drivers / Frameworks / Toolkits (https://www.datasculpting.ai/inference/forumdisplay.php?fid=8)
+--- Forum: PyTorch / JAX / Other Frameworks (https://www.datasculpting.ai/inference/forumdisplay.php?fid=34)
+--- Thread: JAX for inference only with LM Studio (/showthread.php?tid=349)
Pages:
1
2
JAX for inference only with LM Studio - mangowire - 09-18-2026
Did a bit of testing and thought I'd post the result here.
Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring NVFP4.
Curious if anyone else landed in the same place.
Re: JAX for inference only with LM Studio - northcraft90 - 09-18-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Once you know the plain path works, then add the faster path back in piece by piece.
Re: JAX for inference only with LM Studio - southden - 09-18-2026
That matches what I was seeing. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: JAX for inference only with LM Studio - macrostack69 - 09-19-2026
Did a small comparison this afternoon. On B200 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: JAX for inference only with LM Studio - cudacrate - 09-22-2026
Quote:Did a small comparison this afternoon. On B200 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.
I hit almost the same issue last night.
Re: JAX for inference only with LM Studio - macrotinker - 09-25-2026
Thanks, that helps. The pattern around structured output has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: JAX for inference only with LM Studio - marbletrack - 09-26-2026
What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: JAX for inference only with LM Studio - circuitwire58 - 09-29-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: JAX for inference only with LM Studio - routerunner - 09-30-2026
Did a small comparison this afternoon. On 5090 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: JAX for inference only with LM Studio - marbleyard32 - 10-01-2026
Quote:Did a small comparison this afternoon. On 5090 with LM Studio, I saw basically the same behavior. The surprise was that memory pressure showed up befo
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.