synthetic data for support bots for Qwen

[emberquill]

Wanted to compare notes with people doing similar things.

I'm mostly working with Kimi in Ollama on B200. The part I'm trying to understand is structured output.

Not looking for the perfect answer, mostly what worked in real use.

[promptpoint]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? I keep promising myself this is the last rebuild.

[vramforge]

Can you fill in a bit more context? Is this local-only or are you routing anything externally? I keep promising myself this is the last rebuild.

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[routeline23]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[audioforge]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[plainworks]

What exact setup are you running? Are you testing this on mini PC or something else?

[dockworks23]

I hit almost the same issue last night.

[rocmhouse90]

Thanks, that helps. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.