where open models feel surprisingly good

[mellowsupply]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and B200.

Would rather learn from your scars than collect my own all at once.

[cozysmith]

I'd start with the simpler stack.

[macropoint]

same here on my side. I keep promising myself this is the last rebuild.

[audiorunner74]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[sparrowgrid]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Not glamorous advice, but repeatable baselines save a lot of time.

[vramforge]

I think the bottleneck is somewhere else. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

I'm happy to test one more variant if you want a second data point. I keep promising myself this is the last rebuild.

[routearch]

That matches what I was seeing. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[kernelpeak]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[northhall88]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[batchember]

same here on my side.