Posts: 46
Threads: 6
Joined: Sep 2026
Reputation:
0
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about KV cache pressure.
Would rather learn from your scars than collect my own all at once.
I care more about stability than squeezing the last possible tokens/sec out of it.
Posts: 184
Threads: 23
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.
Mostly CUDA, sometimes wrong, usually measured
Posts: 69
Threads: 3
Joined: Sep 2026
Reputation:
0
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
still benchmarking everything
Posts: 60
Threads: 5
Joined: Sep 2026
Reputation:
0
Good point. The pattern around router policy has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Posts: 184
Threads: 23
Joined: Sep 2026
Reputation:
0
I hit almost the same issue last night. I keep promising myself this is the last rebuild.
Mostly CUDA, sometimes wrong, usually measured
Posts: 184
Threads: 23
Joined: Sep 2026
Reputation:
0
Quote:I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Mostly CUDA, sometimes wrong, usually measured
Posts: 34
Threads: 2
Joined: Sep 2026
Reputation:
0
Can you fill in a bit more context? Are you testing this on B200 or something else?
small models can still be good
Posts: 83
Threads: 6
Joined: Sep 2026
Reputation:
0
Not sure I agree there. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Posts: 83
Threads: 6
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece. Turns out 'just one more test' was a lie.
Posts: 50
Threads: 6
Joined: Sep 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec. Not glamorous advice, but repeatable baselines save a lot of time.
