JAX for inference only on dual 3090

[homelabhenry]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and 4090.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

[audioforge]

subscribed, curious where this lands.

I'm happy to test one more variant if you want a second data point.

[sparrowworks88]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

[tinycontext]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.

Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[pixelcrate82]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[deepseeked]

A couple things I'd check first. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[dockember]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

[vrambandit]

Thanks, that helps. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer. The logs were somehow calmer than I was.

[quietarch]

Appreciate the detail. The pattern around HBM constraints has been pretty similar for me.

[thetashed24]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.