This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like quant drift decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about HBM constraints.

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and 4090 felt surprisingly usable once I tuned around KV cache pressure.

Maybe useful as a starting point for newer folks.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and A6000.

Simple and boring is a valid answer.

The part that surprised me was how much the boring details mattered more than the model choice.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on mini PC, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

This is mostly for local work rather than public serving.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and mini PC felt surprisingly usable once I tuned around tail latency.

Feel free to tear it apart if you see something dumb.

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.

Did a bit of testing and thought I'd post the result here.

Short version: vLLM behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like MoE serving decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.