when TensorRT-LLM clearly wins

[routecrate]

that fixed it for me too.

[sparrowcraft]

Good point. I don't think your conclusion is far off.

[rocmriley]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[sparrowhall]

Good point. That matches what I saw on mini PC.

[kernelmike]

I hit almost the same issue last night.

[sysstation69]

I hit almost the same issue last night.

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did. At this point the real bottleneck is my patience.