latency vs throughput for small deployments on dual 3090

[kernelmike]

[quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that move

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time.