Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Wanted to compare notes with people doing similar things.
I'm mostly working with Phi in llama.cpp on B200. The part I'm trying to understand is tool calling.
If you've tested something similar, what did you keep and what did you stop bothering with?
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
throughput matters, but tail latency matters too
Posts: 39
Threads: 1
Joined: Sep 2026
Reputation:
0
Not sure I agree there. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
queue depth watcher
Posts: 53
Threads: 7
Joined: Sep 2026
Reputation:
0
I hit almost the same issue last night.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
I tested this tonight. On 5090 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
throughput matters, but tail latency matters too
Posts: 63
Threads: 5
Joined: Sep 2026
Reputation:
0
That matches what I was seeing. That matches what I saw on 5090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Posts: 41
Threads: 4
Joined: Aug 2026
Reputation:
0
Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Posts: 45
Threads: 3
Joined: Sep 2026
Reputation:
0
subscribed, curious where this lands.
Posts: 51
Threads: 3
Joined: Sep 2026
Reputation:
0
I hit almost the same issue last night.
