consumer GPU serving economics for Gemma

[papermixer4]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on 5090, keeping prefix caching in mind from the start.

Feel free to tear it apart if you see something dumb.

[sparrowshed]

A couple things I'd check first. If you're on Ollama, I'd confirm the version first and then reduce variables until the behavior is obvious.

[rocmpath]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[signalcraft14]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[kernelmike]

I tested this tonight. On 4090 with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did.

[fastscope]

subscribed, curious where this lands.

[edgeriver]

I'd approach it in a boring order. If you're on MLX, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.