quality tests for local assistants with llama.cpp

[torchsupply99]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Curious whether that lines up with what other people are seeing.

[routeline23]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

I'm happy to test one more variant if you want a second data point.

[fusedtinker]

that fixed it for me too.

[embershed99]

that fixed it for me too.

That matches what I was seeing. The pattern around tool calling has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[benchplease]

that fixed it for me too.

Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.

[batchpeak]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.