AI pin style devices but useful?

[routeline23]

Wanted to compare notes with people doing similar things.

My current setup is TensorRT-LLM with 5090, and I keep running into continuous batching.

If you've tested something similar, what did you keep and what did you stop bothering with?

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[routeline23]

same here on my side.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[audiorunner74]

worth checking the driver first.