what are you all running this week on dual 3090

[phasetrack]

I'm trying to sanity-check my setup.

I'm mostly working with Llama in SGLang on 4090. The part I'm trying to understand is prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.

This is mostly for local work rather than public serving.

[plaingarden18]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[marbletrack]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

If you post your exact settings, people here can probably narrow it down pretty fast.

[plaingarden18]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

[ggufwire22]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size.

subscribed, curious where this lands.

[marbletrack]

that fixed it for me too.

[cozygrid12]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[rocmrunner40]

that fixed it for me too.

[turbogrid11]

I hit almost the same issue last night.

[tensorkite]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.