Posts: 48
Threads: 9
Joined: Sep 2026
Reputation:
0
Did a bit of testing and thought I'd post the result here.
After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around HBM constraints.
Happy to post more details if anyone wants them.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Posts: 31
Threads: 3
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
measure twice, benchmark three times
Posts: 49
Threads: 7
Joined: Oct 2026
Reputation:
0
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
trying not to rebuild again
Posts: 55
Threads: 8
Joined: Sep 2026
Reputation:
0
Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Thanks, that helps. That matches what I saw on Threadripper box. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Apple Silicon • MLX mostly
Posts: 40
Threads: 3
Joined: Sep 2026
Reputation:
0
Quote:Quote:Quote:Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.
Apple Silicon • MLX mostly
Posts: 31
Threads: 3
Joined: Sep 2026
Reputation:
0
What exact setup are you running? Is this local-only or are you routing anything externally?
I'm happy to test one more variant if you want a second data point.
Posts: 38
Threads: 4
Joined: Aug 2026
Reputation:
0
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
curious and caffeinated
