consumer GPU serving economics

[sparrowshed]

Did a bit of testing and thought I'd post the result here.

Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.

[cudapeak]

A couple things I'd check first. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives.

[southtinker]

That matches what I was seeing. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[batchember]

A couple things I'd check first. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.

[bytestack]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[bytelab]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

subscribed, curious where this lands.

[kernelrunner]

following this because I'm in the same boat.

I'm happy to test one more variant if you want a second data point.

[rocmhouse90]

I'd start with the simpler stack.

[batchember]

Not sure I agree there. I've seen people blame KV cache pressure when the real issue was much more mundane.