human evals on small teams for Llama

[batchfield55]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[cudacrate]

Interesting, I hadn't tried that. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[oxidemixer]

Appreciate the detail. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vramarch]

same here on my side.

[rasterbench]

My first guess would be this: On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[labpath]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[tinycontext]

Good point. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[turboscope5]

that fixed it for me too.

[paperrunner]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[localcraft17]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?