+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: human evals on small teams (/showthread.php?tid=1341)
Pages:
1
2
human evals on small teams - vectorember - 10-16-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: llama.cpp on GB200, keeping small local stacks in mind from the start.
If someone wants the exact settings I can clean them up and post them.
Re: human evals on small teams - southtinker - 10-18-2026
Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: human evals on small teams - benchplease - 10-18-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: human evals on small teams - paperrunner - 10-18-2026
My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: human evals on small teams - sparrowshed - 10-18-2026
A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: human evals on small teams - oxideworks59 - 10-18-2026
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: human evals on small teams - sparrowcraft - 10-18-2026
Quote:Thanks, that helps. The pattern around long context has been pretty similar for me. I got better results once I stopped chasing one metric and just lo
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: human evals on small teams - audionode8 - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Turns out 'just one more test' was a lie.
Re: human evals on small teams - mintforge - 10-18-2026
Maybe, but I had the opposite result. I get why that seems right, but on 4090 the bottleneck felt elsewhere.
Re: human evals on small teams - macrolab - 10-18-2026
Thanks, that helps. The pattern around long context has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.