human evals on small teams

[emberpeak]

Maybe I'm missing something obvious.

I'm mostly working with Llama in SGLang on dual 3090. The part I'm trying to understand is small local stacks.

If you've tested something similar, what did you keep and what did you stop bothering with?

[batchwire]

I hit almost the same issue last night.

[bytepilot]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[audiodesk]

Ran a quick check on my box. On 4090 with MLX, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[amberhall]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[northgrid33]

Good point. The pattern around batch size has been pretty similar for me.

[vramport]

Interesting, I hadn't tried that. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[thetashed24]

Interesting, I hadn't tried that. That matches what I saw on dual 3090. I got better results once I stopped chasing one metric and just looked at whet

Appreciate the detail. That matches what I saw on M3 Max. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deltastation]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[dockkite69]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. The logs were somehow calmer than I was.