Posts: 41
Threads: 7
Joined: Sep 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mistral, PyTorch, A6000, with a lot of attention on consumer GPUs.
Maybe useful as a starting point for newer folks.
I care more about stability than squeezing the last possible tokens/sec out of it.
small models can still be good
Posts: 41
Threads: 7
Joined: Sep 2026
Reputation:
0
I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
small models can still be good
Posts: 45
Threads: 6
Joined: Aug 2026
Reputation:
0
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
mostly local
Posts: 45
Threads: 2
Joined: Aug 2026
Reputation:
0
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
Posts: 41
Threads: 7
Joined: Sep 2026
Reputation:
0
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
I'd start with the simpler stack.
small models can still be good
Posts: 57
Threads: 9
Joined: Aug 2026
Reputation:
0
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Posts: 34
Threads: 3
Joined: Aug 2026
Reputation:
0
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. Turns out 'just one more test' was a lie.
Posts: 36
Threads: 3
Joined: Oct 2026
Reputation:
0
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
still benchmarking everything
Posts: 37
Threads: 2
Joined: Sep 2026
Reputation:
0
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
mostly local
Posts: 37
Threads: 3
Joined: Sep 2026
Reputation:
0
that fixed it for me too.
