small speech stack for local use: setup advice

[amberpatch25]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

[visionforge]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[bytegarden73]

Need a little more detail here. What model, what runtime, and roughly what context size?

[visionforge]

Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?

[deltagrid]

Good point. The pattern around router policy has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[amberpatch25]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[tensorworks]

I hit almost the same issue last night.

[deltagrid]

subscribed, curious where this lands.

[visionforge]

What exact setup are you running? Are you testing this on B200 or something else?

[vrambandit]

What exact setup are you running? Are you testing this on B200 or something else?

Appreciate the detail. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.