+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use: setup advice (/showthread.php?tid=1195)
Pages:
1
2
Re: small speech stack for local use: setup advice - macropoint - 10-16-2026
I hit almost the same issue last night.
Re: small speech stack for local use: setup advice - cobaltspark - 10-16-2026
Interesting, I hadn't tried that. The pattern around speculative decoding has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: small speech stack for local use: setup advice - visionforge - 10-17-2026
Need a little more detail here. Is this local-only or are you routing anything externally?
Re: small speech stack for local use: setup advice - amberpatch25 - 10-17-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: small speech stack for local use: setup advice - bytegarden73 - 10-17-2026
Quote:Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
I'd approach it in a boring order. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.