Jetson still a thing

[lablab]

I'm trying to sanity-check my setup.

I'm mostly working with Mixtral in PyTorch on mini PC. The part I'm trying to understand is MoE serving.

If you've tested something similar, what did you keep and what did you stop bothering with?

[homelabhenry]

I'm trying to sanity-check my setup.

My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[localpatch]

What exact setup are you running? What model, what runtime, and roughly what context size?

[vramghost]

That matches what I was seeing. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[thetatinker77]

Appreciate the detail. That matches what I saw on GB200.

[batchfield55]

that fixed it for me too.

[vramghost]

Not sure I agree there. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

I'm happy to test one more variant if you want a second data point.

[cobalttinker]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[homelabhenry]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying yo

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece. I keep promising myself this is the last rebuild.

[oxiderunner]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with FLUX and compare from there.

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

If you post your exact settings, people here can probably narrow it down pretty fast.