+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Online / Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=19)
+--- Thread: good hosted AI app that still respects privacy with vLLM (/showthread.php?tid=452)
Pages:
1
2
Re: good hosted AI app that still respects privacy with vLLM - latentcraft - 09-30-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.
Re: good hosted AI app that still respects privacy with vLLM - modrunner27 - 10-01-2026
Quote:Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up b
same here on my side.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: good hosted AI app that still respects privacy with vLLM - torchsmith25 - 10-01-2026
Thanks, that helps. I ended up in about the same place using llama.cpp.
Re: good hosted AI app that still respects privacy with vLLM - tensorriver - 10-02-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: good hosted AI app that still respects privacy with vLLM - thetaarch43 - 10-03-2026
Quote:A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?