Any point in NPUs yet on dual 3090

[lablab]

[quote]Appreciate the detail. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad ass

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[northquill]

Did a small comparison this afternoon. On dual 3090 with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[batchember]

Good point. I don't think your conclusion is far off.

[cobaltworks25]

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

[labnest]

Appreciate the detail. The pattern around embeddings has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.