Any point in NPUs yet on dual 3090

[thetashed24]

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

Would appreciate practical advice rather than theory.

[batchwire]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal.

[cudahouse]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[bytehall]

Interesting, I hadn't tried that. That matches what I saw on B200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cudahouse]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. Not glamorous advice, but repeatable baselines save a lot of time.

[latentscope35]

worth checking the driver first.

[batchsize]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

I'm happy to test one more variant if you want a second data point.

[batchyard]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[cudacrate]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[cacheline]

Appreciate the detail. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.