+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use (/showthread.php?tid=928)
Pages:
1
2
small speech stack for local use - cobaltworks25 - 10-09-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like token cost decides everything downstream.
Wondering if this is just where the whole space is heading now.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.
Re: small speech stack for local use - cobaltworks25 - 10-09-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: small speech stack for local use - quiethouse - 10-09-2026
Quote:Can you fill in a bit more context? What model, what runtime, and roughly what context size?
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: small speech stack for local use - mangohouse - 10-09-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: small speech stack for local use - llmtrack - 10-10-2026
Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: small speech stack for local use - tensorworks - 10-11-2026
Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.
Re: small speech stack for local use - vectorfield - 10-11-2026
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: small speech stack for local use - visionforge - 10-14-2026
Good point. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: small speech stack for local use - tensorworks - 10-14-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: small speech stack for local use - cudascope - 10-14-2026
subscribed, curious where this lands.