+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Embedding / Reranking Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=53)
+--- Thread: tool calling reliability with PyTorch (/showthread.php?tid=1259)
Pages:
1
2
tool calling reliability with PyTorch - fusedtinker - 10-16-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around consumer GPUs.
If someone wants the exact settings I can clean them up and post them.
This is mostly for local work rather than public serving.
Re: tool calling reliability with PyTorch - phasestream53 - 10-17-2026
Appreciate the detail. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: tool calling reliability with PyTorch - rocmhouse90 - 10-18-2026
same here on my side.
There are a few different conversations hiding inside this thread, so being explicit about the goal helps.
Re: tool calling reliability with PyTorch - turbogrid11 - 10-18-2026
same here on my side.
Re: tool calling reliability with PyTorch - mangopath4 - 10-18-2026
Ran a quick check on my box. On A6000 with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.
Re: tool calling reliability with PyTorch - torchtrack24 - 10-18-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there.
Re: tool calling reliability with PyTorch - fastlab - 10-18-2026
That matches what I was seeing. That matches what I saw on B200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: tool calling reliability with PyTorch - griffinloom66 - 10-18-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: tool calling reliability with PyTorch - routecrate - 10-19-2026
worth checking the driver first.
Re: tool calling reliability with PyTorch - forgemixer - 10-19-2026
I tested this tonight. On B200 with Ollama, I saw basically the same behavior. The main thing that moved was latency, not quality.