+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Online / Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=19)
+--- Thread: good hosted AI app that still respects privacy with SGLang (/showthread.php?tid=729)
good hosted AI app that still respects privacy with SGLang - torchsmith25 - 10-03-2026
This keeps coming up in conversation so I figured I'd start a thread.
One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.
Wondering if this is just where the whole space is heading now.
Re: good hosted AI app that still respects privacy with SGLang - vramarch - 10-03-2026
Quote:This keeps coming up in conversation so I figured I'd start a thread.
same here on my side.
Re: good hosted AI app that still respects privacy with SGLang - llm_wren - 10-04-2026
I think the bottleneck is somewhere else. I've seen people blame consumer GPUs when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: good hosted AI app that still respects privacy with SGLang - emberquill - 10-04-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: good hosted AI app that still respects privacy with SGLang - sparrowshed - 10-05-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal.
Re: good hosted AI app that still respects privacy with SGLang - rocmpath - 10-06-2026
I'd start with the simpler stack.
Re: good hosted AI app that still respects privacy with SGLang - amberplane - 10-07-2026
Not sure I agree there. I've seen people blame batch size when the real issue was much more mundane.
Re: good hosted AI app that still respects privacy with SGLang - emberrunner - 10-07-2026
subscribed, curious where this lands.
Re: good hosted AI app that still respects privacy with SGLang - modbench - 10-07-2026
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: good hosted AI app that still respects privacy with SGLang - embershed99 - 10-07-2026
Good point. The pattern around token cost has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.