+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: human evals on small teams (/showthread.php?tid=274)
human evals on small teams - fusedtinker - 09-14-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of SGLang and mini PC felt surprisingly usable once I tuned around tokens/sec.
Feel free to tear it apart if you see something dumb.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: human evals on small teams - signalcraft14 - 09-16-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: human evals on small teams - thetashed24 - 09-16-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefill/decode split. Once you know the plain path works, then add the faster path back in piece by piece.
Re: human evals on small teams - rasterhall - 09-16-2026
Thanks, that helps. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: human evals on small teams - kernelhouse70 - 09-17-2026
Can you fill in a bit more context? Are you testing this on 5090 or something else?
Re: human evals on small teams - cachelab - 09-17-2026
Quote:Can you fill in a bit more context? Are you testing this on 5090 or something else?
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.