clean API wrappers for local agents: early impressions

[localcraft17]

Wanted to compare notes with people doing similar things.

I'm mostly working with Qwen in Ollama on 4090. The part I'm trying to understand is KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.

[audionode8]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[emberrunner]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[rocmriley]

I'd approach it in a boring order. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[thetafield]

I'd start with the simpler stack.

[routerunner]

same here on my side.

[ggufvault]

subscribed, curious where this lands.

[phaseshed]

That matches what I was seeing. The pattern around consumer GPUs has been pretty similar for me.

[vramghost]

Interesting, I hadn't tried that. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mangopath4]

Thanks, that helps. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.