Posts: 341
Threads: 51
Joined: Sep 2026
Reputation:
0
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of PyTorch and M3 Max felt surprisingly usable once I tuned around KV cache pressure.
If someone wants the exact settings I can clean them up and post them.
measure twice, benchmark three times
Posts: 53
Threads: 7
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Posts: 168
Threads: 14
Joined: Sep 2026
Reputation:
0
I tested this tonight. On 5090 with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.
2x3090 • llama.cpp • too much VRAM is still not enough
Posts: 62
Threads: 5
Joined: Sep 2026
Reputation:
0
Interesting, I hadn't tried that. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
trying not to rebuild again
Posts: 137
Threads: 16
Joined: Sep 2026
Reputation:
0
My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
throughput matters, but tail latency matters too
Posts: 44
Threads: 3
Joined: Oct 2026
Reputation:
0
Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
small models can still be good
