I'm trying to sanity-check my setup.

I'm mostly working with Llama in SGLang on 4090. The part I'm trying to understand is prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.

This is mostly for local work rather than public serving.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and GB200 felt surprisingly usable once I tuned around router policy.

If someone wants the exact settings I can clean them up and post them.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like VRAM headroom decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and A6000.

Would rather learn from your scars than collect my own all at once.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like reranking decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

Did a bit of testing and thought I'd post the result here.

After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around quant drift.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Curious whether that lines up with what other people are seeing.

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using vLLM on B200. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and llama.cpp for FLUX. The interesting difference for me has been consumer GPUs.

Trying to avoid rebuilding the whole thing twice.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?