Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, vLLM, 5090, with a lot of attention on structured output.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tokens/sec decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like workstation thermals decides everything downstream.

Wondering if this is just where the whole space is heading now.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months? At this point the real bottleneck is my patience.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, MLX, 7900 XTX, with a lot of attention on INT4.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and dual 3090.

Would rather learn from your scars than collect my own all at once.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Maybe obvious to some of you, but it clicked for me this week.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and mini PC.

Would rather learn from your scars than collect my own all at once.