+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Apps / Utilities / Tools (https://www.datasculpting.ai/inference/forumdisplay.php?fid=18)
+--- Forum: Online / Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=19)
+--- Thread: good hosted AI app that still respects privacy (/showthread.php?tid=59)
Pages:
1
2
good hosted AI app that still respects privacy - dockworks23 - 08-28-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of llama.cpp and Threadripper box felt surprisingly usable once I tuned around token cost.
Feel free to tear it apart if you see something dumb.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: good hosted AI app that still respects privacy - macrostack69 - 08-28-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: good hosted AI app that still respects privacy - thetarunner - 08-28-2026
same here on my side.
Re: good hosted AI app that still respects privacy - batchshed - 08-28-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.
Re: good hosted AI app that still respects privacy - bytelab - 08-31-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.
Thanks, that helps. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: good hosted AI app that still respects privacy - batchshed - 08-31-2026
Quote:[quote]What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/sec.[/q
I'd approach it in a boring order. If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: good hosted AI app that still respects privacy - vramforge - 09-03-2026
Quote:[quote][quote]What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tokens/
That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: good hosted AI app that still respects privacy - bytelab - 09-05-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: good hosted AI app that still respects privacy - mangopath4 - 09-05-2026
A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: good hosted AI app that still respects privacy - mangopeak72 - 09-07-2026
I'd start with the simpler stack.