Posts: 43
Threads: 6
Joined: Aug 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of llama.cpp and 4090 felt surprisingly usable once I tuned around structured output.
Feel free to tear it apart if you see something dumb.
learning in public
Posts: 51
Threads: 3
Joined: Sep 2026
Reputation:
0
Good point. That matches what I saw on 5090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Posts: 63
Threads: 5
Joined: Sep 2026
Reputation:
0
What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Posts: 49
Threads: 5
Joined: Sep 2026
Reputation:
0
What exact setup are you running? Are you testing this on A6000 or something else?
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Quote:What exact setup are you running? Are you testing this on A6000 or something else?
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
throughput matters, but tail latency matters too
Posts: 59
Threads: 7
Joined: Oct 2026
Reputation:
0
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
learning in public
