Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and 4090 felt surprisingly usable once I tuned around continuous batching.

Feel free to tear it apart if you see something dumb.

I'm trying to sanity-check my setup.

I'm mostly working with Gemma in MLX on mini PC. The part I'm trying to understand is quant drift.

Would appreciate practical advice rather than theory.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on A6000, keeping prefix caching in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

Short version: SGLang behaved well for me, but only after I stopped ignoring router policy.

Nothing scientific here, just practical notes from a real setup.

This is mostly for local work rather than public serving.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and GB200.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like batch size decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like batch size decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.