+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Proprietary / Frontier (https://www.datasculpting.ai/inference/forumdisplay.php?fid=48)
+--- Thread: where open models feel surprisingly good (/showthread.php?tid=206)
Pages:
1
2
where open models feel surprisingly good - mellowsupply - 09-09-2026
Posting this in case it saves someone else a little time.
I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and B200.
Would rather learn from your scars than collect my own all at once.
Re: where open models feel surprisingly good - cozysmith - 09-09-2026
I'd start with the simpler stack.
Re: where open models feel surprisingly good - macropoint - 09-10-2026
same here on my side. I keep promising myself this is the last rebuild.
Re: where open models feel surprisingly good - audiorunner74 - 09-10-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: where open models feel surprisingly good - sparrowgrid - 09-12-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Not glamorous advice, but repeatable baselines save a lot of time.
Re: where open models feel surprisingly good - vramforge - 09-12-2026
I think the bottleneck is somewhere else. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
I'm happy to test one more variant if you want a second data point. I keep promising myself this is the last rebuild.
Re: where open models feel surprisingly good - routearch - 09-12-2026
That matches what I was seeing. I ended up in about the same place using llama.cpp. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: where open models feel surprisingly good - kernelpeak - 09-12-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: where open models feel surprisingly good - northhall88 - 09-12-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Re: where open models feel surprisingly good - batchember - 09-13-2026
same here on my side.