+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Online / Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=19)
+--- Thread: good hosted AI app that still respects privacy with vLLM (/showthread.php?tid=452)
Pages:
1
2
good hosted AI app that still respects privacy with vLLM - macronest - 09-23-2026
Spent some time on this today and wanted to compare notes.
I've gone back and forth between LM Studio and vLLM for DeepSeek. The interesting difference for me has been HBM constraints.
Trying to avoid rebuilding the whole thing twice.
I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.
Re: good hosted AI app that still respects privacy with vLLM - macronest - 09-24-2026
Appreciate the detail. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: good hosted AI app that still respects privacy with vLLM - macrostack69 - 09-24-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: good hosted AI app that still respects privacy with vLLM - sparrowshed - 09-24-2026
That matches what I was seeing. The pattern around small local stacks has been pretty similar for me.
Re: good hosted AI app that still respects privacy with vLLM - thetarunner - 09-24-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: good hosted AI app that still respects privacy with vLLM - modrunner27 - 09-27-2026
Interesting, I hadn't tried that. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: good hosted AI app that still respects privacy with vLLM - northgrid33 - 09-27-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs.
Re: good hosted AI app that still respects privacy with vLLM - deepseeked - 09-29-2026
worth checking the driver first.
Re: good hosted AI app that still respects privacy with vLLM - emberrunner - 09-30-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.
Re: good hosted AI app that still respects privacy with vLLM - deepseeked - 09-30-2026
Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.