tool calling reliability with PyTorch

[fusedtinker]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around consumer GPUs.

If someone wants the exact settings I can clean them up and post them.

This is mostly for local work rather than public serving.

[phasestream53]

Appreciate the detail. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[rocmhouse90]

same here on my side.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[turbogrid11]

same here on my side.

[mangopath4]

Ran a quick check on my box. On A6000 with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[torchtrack24]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.

[fastlab]

That matches what I was seeing. That matches what I saw on B200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[griffinloom66]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[routecrate]

worth checking the driver first.

[forgemixer]

I tested this tonight. On B200 with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality.