Posts: 37
Threads: 6
Joined: Oct 2026
Reputation:
0
Did a bit of testing and thought I'd post the result here.
Short version: llama.cpp behaved well for me, but only after I stopped ignoring tokens/sec.
Nothing scientific here, just practical notes from a real setup.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Posts: 28
Threads: 5
Joined: Sep 2026
Reputation:
0
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Posts: 121
Threads: 15
Joined: Sep 2026
Reputation:
0
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
image models by night, local pipelines by day
Posts: 31
Threads: 2
Joined: Oct 2026
Reputation:
0
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Posts: 41
Threads: 4
Joined: Aug 2026
Reputation:
0
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Posts: 121
Threads: 15
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
image models by night, local pipelines by day
Posts: 52
Threads: 4
Joined: Sep 2026
Reputation:
0
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
Quote:I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before th
What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
throughput matters, but tail latency matters too
Posts: 48
Threads: 8
Joined: Oct 2026
Reputation:
0
Interesting, I hadn't tried that. That matches what I saw on M3 Max. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Posts: 40
Threads: 8
Joined: Sep 2026
Reputation:
0
Ran a quick check on my box. On mini PC with PyTorch, I saw basically the same behavior. The main thing that moved was latency, not quality.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
