+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: human evals on small teams (/showthread.php?tid=858)
Pages:
1
2
human evals on small teams - emberpeak - 10-07-2026
Maybe I'm missing something obvious.
I'm mostly working with Llama in SGLang on dual 3090. The part I'm trying to understand is small local stacks.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: human evals on small teams - batchwire - 10-08-2026
I hit almost the same issue last night.
Re: human evals on small teams - bytepilot - 10-08-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: human evals on small teams - audiodesk - 10-09-2026
Ran a quick check on my box. On 4090 with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: human evals on small teams - amberhall - 10-10-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: human evals on small teams - northgrid33 - 10-10-2026
Good point. The pattern around batch size has been pretty similar for me.
Re: human evals on small teams - vramport - 10-10-2026
Interesting, I hadn't tried that. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: human evals on small teams - thetashed24 - 10-10-2026
Quote:Interesting, I hadn't tried that. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whet
Appreciate the detail. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: human evals on small teams - deltastation - 10-11-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: human evals on small teams - dockkite69 - 10-14-2026
Quote:Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. The logs were somehow calmer than I was.