Did a bit of testing and thought I'd post the result here.

Short version: vLLM behaved well for me, but only after I stopped ignoring structured output.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on dual 3090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Curious whether that lines up with what other people are seeing.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once tokens/sec becomes important.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding. The logs were somehow calmer than I was.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tail latency decides everything downstream.

Curious whether that lines up with what other people are seeing.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Curious whether that lines up with what other people are seeing.

The part that surprised me was how much the boring details mattered more than the model choice.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and M3 Max.

I'm especially interested in mistakes worth avoiding.

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, LM Studio, A6000, with a lot of attention on INT4.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.