human evals on small teams

[vectorember]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on GB200, keeping small local stacks in mind from the start.

If someone wants the exact settings I can clean them up and post them.

[southtinker]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[benchplease]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[paperrunner]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[sparrowshed]

A couple things I'd check first. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

[oxideworks59]

Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[sparrowcraft]

Thanks, that helps. The pattern around long context has been pretty similar for me. I got better results once I stopped chasing one metric and just lo

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[audionode8]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding. Turns out 'just one more test' was a lie.

[mintforge]

Maybe, but I had the opposite result. I get why that seems right, but on 4090 the bottleneck felt elsewhere.

[macrolab]

Thanks, that helps. The pattern around long context has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

If you post your exact settings, people here can probably narrow it down pretty fast.