good hosted AI app that still respects privacy with vLLM

[latentcraft]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[modrunner27]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up b

same here on my side.

If you post your exact settings, people here can probably narrow it down pretty fast.

[torchsmith25]

Thanks, that helps. I ended up in about the same place using llama.cpp.

[tensorriver]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[thetaarch43]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?