+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: human evals on small teams for Gemma (/showthread.php?tid=226)
Pages:
1
2
human evals on small teams for Gemma - plainscope50 - 09-12-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Simple and boring is a valid answer.
Re: human evals on small teams for Gemma - torchsupply99 - 09-12-2026
Interesting, I hadn't tried that. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: human evals on small teams for Gemma - turboyard94 - 09-13-2026
Quote:Interesting, I hadn't tried that. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked
following this because I'm in the same boat.
Re: human evals on small teams for Gemma - batchscope72 - 09-14-2026
Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.
Re: human evals on small teams for Gemma - localpatch78 - 09-14-2026
What worked for me was pretty simple. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: human evals on small teams for Gemma - audioriver - 09-14-2026
A couple questions before I guess too hard. Are you testing this on A6000 or something else?
Re: human evals on small teams for Gemma - mangoquill - 09-14-2026
that fixed it for me too.
Re: human evals on small teams for Gemma - phaseport36 - 09-15-2026
A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: human evals on small teams for Gemma - sparrowcraft - 09-16-2026
That matches what I was seeing. I ended up in about the same place using SGLang.
Re: human evals on small teams for Gemma - routearch - 09-16-2026
Quote:That matches what I was seeing. I ended up in about the same place using SGLang.
Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.