small speech stack for local use with LM Studio

[cobaltspark]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[quiethouse]

Maybe, but I had the opposite result. I've seen people blame continuous batching when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[llmtrack]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

[kernelpeak]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[tensorworks]

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[localcraft17]

I'd start with the simpler stack.