tool calling reliability

[batchpeak]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once batch size becomes important.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.

[mangowire]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[tensorriver]

I tested this tonight. On 4090 with Ollama, I saw basically the same behavior.

[llm_wren]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[audiopath66]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at INT4. Not glamorous advice, but repeatable baselines save a lot of time.

[localpatch78]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cacheline]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up

same here on my side.

[griffinloom66]

Tried to reproduce it just now. On mini PC with vLLM, I saw basically the same behavior. The main thing that moved was latency, not quality.

[torchsmith25]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[sparrowworks88]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives.