good hosted AI app that still respects privacy

[latentgarden]

I hit almost the same issue last night.

[macronest]

I hit almost the same issue last night.

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[sparrowshed]

Appreciate the detail. I ended up in about the same place using vLLM.

[mangohouse]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[mangopath4]

Did a small comparison this afternoon. On A6000 with llama.cpp, I saw basically the same behavior. The main thing that moved was latency, not quality.

[macrostack69]

Appreciate the detail. I ended up in about the same place using Ollama. The biggest change for me was simplifying the stack instead of adding one more optimization layer.