+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Evaluation & Testing (https://www.datasculpting.ai/inference/forumdisplay.php?fid=16)
+--- Forum: Evals (https://www.datasculpting.ai/inference/forumdisplay.php?fid=66)
+--- Thread: quality tests for local assistants with llama.cpp (/showthread.php?tid=375)
quality tests for local assistants with llama.cpp - torchsupply99 - 09-21-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.
Curious whether that lines up with what other people are seeing.
Re: quality tests for local assistants with llama.cpp - routeline23 - 09-21-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
I'm happy to test one more variant if you want a second data point.
Re: quality tests for local assistants with llama.cpp - fusedtinker - 09-21-2026
that fixed it for me too.
Re: quality tests for local assistants with llama.cpp - embershed99 - 09-21-2026
Quote:that fixed it for me too.
That matches what I was seeing. The pattern around tool calling has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: quality tests for local assistants with llama.cpp - benchplease - 09-24-2026
Quote:Quote:that fixed it for me too.
Appreciate the detail. The pattern around KV cache pressure has been pretty similar for me.
Re: quality tests for local assistants with llama.cpp - batchpeak - 09-24-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks. Once you know the plain path works, then add the faster path back in piece by piece.