good hosted AI app that still respects privacy with vLLM

[macronest]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and vLLM for DeepSeek. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.

[macronest]

Appreciate the detail. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[macrostack69]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[sparrowshed]

That matches what I was seeing. The pattern around small local stacks has been pretty similar for me.

[thetarunner]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[modrunner27]

Interesting, I hadn't tried that. That matches what I saw on dual 3090. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[northgrid33]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs.

[deepseeked]

worth checking the driver first.

[emberrunner]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[deepseeked]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.