+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use with SGLang (/showthread.php?tid=122)
small speech stack for local use with SGLang - macropoint - 09-04-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.
Maybe obvious to some of you, but it clicked for me this week.
Re: small speech stack for local use with SGLang - stackarch - 09-04-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: small speech stack for local use with SGLang - cobaltspark - 09-07-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: small speech stack for local use with SGLang - cobaltworks25 - 09-07-2026
What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: small speech stack for local use with SGLang - oxidegrove40 - 09-08-2026
A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: small speech stack for local use with SGLang - deltastation - 09-08-2026
I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: small speech stack for local use with SGLang - phasescope - 09-09-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.
Re: small speech stack for local use with SGLang - mangowire - 09-09-2026
Interesting, I hadn't tried that. The pattern around continuous batching has been pretty similar for me.
Re: small speech stack for local use with SGLang - dockriver - 09-09-2026
Can you fill in a bit more context? Are you testing this on GB200 or something else?