when TensorRT-LLM clearly wins

[macrotinker]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and B200.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[kernelmike]

I think the bottleneck is somewhere else. I've seen people blame consumer GPUs when the real issue was much more mundane. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[vramscope]

worth checking the driver first.

[kernelmike]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[kernelgrid]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[kernelmike]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[syshouse]

What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[audioforge]

Interesting, I hadn't tried that. The pattern around token cost has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

If you post your exact settings, people here can probably narrow it down pretty fast.

[pixelcrate82]

Thanks, that helps. The pattern around router policy has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.