small speech stack for local use: what are you seeing?

[amberpatch25]

Did a bit of testing and thought I'd post the result here.

Short version: MLX behaved well for me, but only after I stopped ignoring prefill/decode split.

Happy to post more details if anyone wants them.

This is mostly for local work rather than public serving.

[cobaltworks25]

Appreciate the detail. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[visionforge]

What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmtrack]

I hit almost the same issue last night.

[cudascope]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[kernelpeak]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[visionforge]

Need a little more detail here. Is this local-only or are you routing anything externally?

[vectorfield]

I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[amberpatch25]

A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[visionforge]

Not sure I agree there. I've seen people blame tool calling when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.