+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: human evals on small teams for Llama (/showthread.php?tid=792)
Pages:
1
2
human evals on small teams for Llama - batchfield55 - 10-06-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like prefix caching decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
I care more about stability than squeezing the last possible tokens/sec out of it.
I expected the biggest difference to come from the model, but the stack around it mattered at least as much.
Re: human evals on small teams for Llama - cudacrate - 10-06-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: human evals on small teams for Llama - oxidemixer - 10-09-2026
Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: human evals on small teams for Llama - vramarch - 10-09-2026
same here on my side.
Re: human evals on small teams for Llama - rasterbench - 10-09-2026
My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: human evals on small teams for Llama - labpath - 10-09-2026
A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: human evals on small teams for Llama - tinycontext - 10-12-2026
Good point. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: human evals on small teams for Llama - turboscope5 - 10-13-2026
that fixed it for me too.
Re: human evals on small teams for Llama - paperrunner - 10-14-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: human evals on small teams for Llama - localcraft17 - 10-14-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?