+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use: setup advice (/showthread.php?tid=97)
Pages:
1
2
small speech stack for local use: setup advice - visionforge - 09-01-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of Ollama and Threadripper box felt surprisingly usable once I tuned around tokens/sec.
Feel free to tear it apart if you see something dumb.
I care more about stability than squeezing the last possible tokens/sec out of it.
Re: small speech stack for local use: setup advice - vramarch - 09-02-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. Once you know the plain path works, then add the faster path back in piece by piece.
Re: small speech stack for local use: setup advice - amberpatch25 - 09-03-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: small speech stack for local use: setup advice - visionforge - 09-03-2026
Quote:My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: small speech stack for local use: setup advice - vectorfield - 09-06-2026
Quote:[quote]My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: small speech stack for local use: setup advice - visionforge - 09-06-2026
worth checking the driver first.
Re: small speech stack for local use: setup advice - quiethouse - 09-09-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: small speech stack for local use: setup advice - mlxgarden - 09-09-2026
Good point. That matches what I saw on dual 3090.
Re: small speech stack for local use: setup advice - cobaltspark - 09-10-2026
A couple things I'd check first. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: small speech stack for local use: setup advice - cobaltworks25 - 09-10-2026
I think the bottleneck is somewhere else. I've seen people blame speculative decoding when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.