how much orchestration is too much: early impressions

[llm_wren]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and Ollama for Kimi. The interesting difference for me has been router policy.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[llm_wren]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[southden]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[quiethouse]

[quote]What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[labline]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[northgrid33]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher.

[macronest]

Ran a quick check on my box. On 4090 with vLLM, I saw basically the same behavior. The main thing that moved was latency, not quality.