+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use (/showthread.php?tid=979)
small speech stack for local use - visionforge - 10-09-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.
What would you pick if you had to live with it for a few months?
Re: small speech stack for local use - bytegarden73 - 10-10-2026
Not sure I agree there. I've seen people blame prefix caching when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: small speech stack for local use - cudascope - 10-10-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: small speech stack for local use - visionforge - 10-10-2026
Good point. The pattern around MoE serving has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: small speech stack for local use - macropoint - 10-10-2026
I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane.
Re: small speech stack for local use - bytegarden73 - 10-10-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: small speech stack for local use - vectorfield - 10-10-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: small speech stack for local use - kernelpeak - 10-11-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: small speech stack for local use - deltastation - 10-11-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: small speech stack for local use - quiethouse - 10-11-2026
Need a little more detail here. Is this local-only or are you routing anything externally?