small speech stack for local use with SGLang

[macropoint]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Maybe obvious to some of you, but it clicked for me this week.

[stackarch]

This keeps coming up in conversation so I figured I'd start a thread.

I'd approach it in a boring order. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[cobaltspark]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[cobaltworks25]

What worked for me was pretty simple. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[oxidegrove40]

A couple things I'd check first. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[deltastation]

I'd approach it in a boring order. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[phasescope]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[mangowire]

Interesting, I hadn't tried that. The pattern around continuous batching has been pretty similar for me.

[dockriver]

Can you fill in a bit more context? Are you testing this on GB200 or something else?