Posts: 30
Threads: 4
Joined: Sep 2026
Reputation:
0
Has anyone else run into this?
I'm mostly working with Mixtral in TensorRT-LLM on GB200. The part I'm trying to understand is batch size.
Would appreciate practical advice rather than theory.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
learning in public
Posts: 53
Threads: 5
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. The pattern around tokens/sec has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
small models can still be good
Posts: 24
Threads: 2
Joined: Aug 2026
Reputation:
0
Interesting, I hadn't tried that. The pattern around batch size has been pretty similar for me.
Posts: 23
Threads: 2
Joined: Aug 2026
Reputation:
0
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Posts: 39
Threads: 1
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
queue depth watcher
Posts: 44
Threads: 3
Joined: Sep 2026
Reputation:
0
Appreciate the detail. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.
Posts: 168
Threads: 14
Joined: Sep 2026
Reputation:
0
Quote:Appreciate the detail. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the wor
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
2x3090 • llama.cpp • too much VRAM is still not enough
Posts: 184
Threads: 23
Joined: Sep 2026
Reputation:
0
I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
Mostly CUDA, sometimes wrong, usually measured
