Posts: 50
Threads: 6
Joined: Sep 2026
Reputation:
0
Did a bit of testing and thought I'd post the result here.
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.
Curious if anyone else landed in the same place.
This is mostly for local work rather than public serving.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
throughput matters, but tail latency matters too
Posts: 32
Threads: 5
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Quote:I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Good point. That matches what I saw on 7900 XTX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
throughput matters, but tail latency matters too
Posts: 54
Threads: 8
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
I tested this tonight. On mini PC with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
throughput matters, but tail latency matters too
Posts: 59
Threads: 7
Joined: Oct 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.
learning in public
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
If you post your exact settings, people here can probably narrow it down pretty fast.
throughput matters, but tail latency matters too
