recipes for a good local stack

[rocmriley]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and mini PC felt surprisingly usable once I tuned around prefill/decode split.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[mlxgarden]

Thanks, that helps. I ended up in about the same place using SGLang.

[rocmriley]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[minthall]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[kernelarch]

Good point. That matches what I saw on mini PC.

[mintquill]

Did a small comparison this afternoon. On A6000 with vLLM, I saw basically the same behavior. I expected the model choice to matter more than it did.

[rocmriley]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[cobalttinker]

Maybe, but I had the opposite result. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[rocmriley]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.

[kernelghost]

Appreciate the detail. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.