Posts: 25
Threads: 4
Joined: Aug 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, 5090, with a lot of attention on prefill/decode split.
Feel free to tear it apart if you see something dumb.
small models can still be good
Posts: 32
Threads: 2
Joined: Sep 2026
Reputation:
0
subscribed, curious where this lands.
Posts: 44
Threads: 3
Joined: Oct 2026
Reputation:
0
Interesting, I hadn't tried that. That matches what I saw on M3 Max. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Posts: 32
Threads: 4
Joined: Aug 2026
Reputation:
0
I hit almost the same issue last night.
curious and caffeinated
Posts: 34
Threads: 5
Joined: Aug 2026
Reputation:
0
Good point. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. I keep promising myself this is the last rebuild.
Posts: 39
Threads: 2
Joined: Sep 2026
Reputation:
0
Maybe, but I had the opposite result. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere.
Posts: 27
Threads: 5
Joined: Sep 2026
Reputation:
0
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
I'm happy to test one more variant if you want a second data point.
Posts: 45
Threads: 4
Joined: Aug 2026
Reputation:
0
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at batch size. Once you know the plain path works, then add the faster path back in piece by piece.
Posts: 85
Threads: 12
Joined: Aug 2026
Reputation:
0
same here on my side.
LoRA tinkerer • dataset janitor
