+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Generative AI (https://www.datasculpting.ai/inference/forumdisplay.php?fid=17)
+--- Forum: Image (https://www.datasculpting.ai/inference/forumdisplay.php?fid=69)
+--- Thread: small speech stack for local use: setup advice (/showthread.php?tid=1195)
Pages:
1
2
small speech stack for local use: setup advice - amberpatch25 - 10-14-2026
Posting this in case it saves someone else a little time.
If you were starting from scratch today for this use case, what would you choose first and what would you ignore?
Would rather learn from your scars than collect my own all at once.
Re: small speech stack for local use: setup advice - visionforge - 10-14-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: small speech stack for local use: setup advice - bytegarden73 - 10-14-2026
Need a little more detail here. What model, what runtime, and roughly what context size?
Re: small speech stack for local use: setup advice - visionforge - 10-14-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: small speech stack for local use: setup advice - deltagrid - 10-14-2026
Good point. The pattern around router policy has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: small speech stack for local use: setup advice - amberpatch25 - 10-14-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: small speech stack for local use: setup advice - tensorworks - 10-15-2026
I hit almost the same issue last night.
Re: small speech stack for local use: setup advice - deltagrid - 10-15-2026
subscribed, curious where this lands.
Re: small speech stack for local use: setup advice - visionforge - 10-15-2026
What exact setup are you running? Are you testing this on B200 or something else?
Re: small speech stack for local use: setup advice - vrambandit - 10-16-2026
Quote:What exact setup are you running? Are you testing this on B200 or something else?
Appreciate the detail. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.