+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use: what are you seeing? (/showthread.php?tid=268)
Pages:
1
2
small speech stack for local use: what are you seeing? - amberpatch25 - 09-13-2026
Did a bit of testing and thought I'd post the result here.
Short version: MLX behaved well for me, but only after I stopped ignoring prefill/decode split.
Happy to post more details if anyone wants them.
This is mostly for local work rather than public serving.
Re: small speech stack for local use: what are you seeing? - cobaltworks25 - 09-14-2026
Appreciate the detail. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: small speech stack for local use: what are you seeing? - visionforge - 09-14-2026
What worked for me was pretty simple. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: small speech stack for local use: what are you seeing? - llmtrack - 09-16-2026
I hit almost the same issue last night.
Re: small speech stack for local use: what are you seeing? - cudascope - 09-17-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?
Re: small speech stack for local use: what are you seeing? - kernelpeak - 09-20-2026
I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: small speech stack for local use: what are you seeing? - visionforge - 09-20-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: small speech stack for local use: what are you seeing? - vectorfield - 09-20-2026
I'd approach it in a boring order. On GB200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: small speech stack for local use: what are you seeing? - amberpatch25 - 09-22-2026
A couple things I'd check first. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: small speech stack for local use: what are you seeing? - visionforge - 09-22-2026
Not sure I agree there. I've seen people blame tool calling when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.