Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
I'm trying to sanity-check my setup.
My current setup is TensorRT-LLM with 4090, and I keep running into continuous batching.
If you've tested something similar, what did you keep and what did you stop bothering with?
This is mostly for local work rather than public serving.
measure twice, benchmark three times
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
Quote:I'm trying to sanity-check my setup.
Not sure I agree there. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.
measure twice, benchmark three times
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
measure twice, benchmark three times
Posts: 23
Threads: 8
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
trying not to rebuild again
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
Quote:Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
measure twice, benchmark three times
Posts: 83
Threads: 6
Joined: Sep 2026
Reputation:
0
Need a little more detail here. Is this local-only or are you routing anything externally?
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
measure twice, benchmark three times
Posts: 121
Threads: 15
Joined: Sep 2026
Reputation:
0
Quote:My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of th
worth checking the driver first.
image models by night, local pipelines by day
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
measure twice, benchmark three times
