setup notes worth sharing: what are you seeing?

[mangoquill]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

[batchshed]

I'd approach it in a boring order. On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[nanobrook]

worth checking the driver first. Turns out 'just one more test' was a lie.

[llmpeak]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[relayforge]

Thanks, that helps. I don't think your conclusion is far off.

[mangoquill]

Ran a quick check on my box. On mini PC with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.

[mlxgarden]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmhouse90]

My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

If you post your exact settings, people here can probably narrow it down pretty fast.

[audioriver]

I tested this tonight. On 7900 XTX with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.