Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on B200, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once workstation thermals becomes important.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and A6000.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.

The part that surprised me was how much the boring details mattered more than the model choice.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around LM Studio and mini PC.

Would rather learn from your scars than collect my own all at once.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 4090.

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it.

Did a bit of testing and thought I'd post the result here.

I tried this with Qwen using TensorRT-LLM on 5090. The headline result was better than I expected, but reranking still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, MLX, B200, with a lot of attention on token cost.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day. Turns out 'just one more test' was a lie.

Did a bit of testing and thought I'd post the result here.

After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.

Curious if anyone else landed in the same place.