Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and vLLM for Kimi. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.

Wanted to compare notes with people doing similar things.

I'm mostly working with Mixtral in MLX on B200. The part I'm trying to understand is tokens/sec.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like token cost decides everything downstream.

Wondering if this is just where the whole space is heading now.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, SGLang, B200, with a lot of attention on VRAM headroom.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and mini PC felt surprisingly usable once I tuned around long context.

Maybe useful as a starting point for newer folks.

I figured I'd ask here before changing everything.

I'm mostly working with DeepSeek in SGLang on Threadripper box. The part I'm trying to understand is prefill/decode split.

Would appreciate practical advice rather than theory.

Curious what people here are seeing.

My current setup is TensorRT-LLM with 5090, and I keep running into KV cache pressure.

Would appreciate practical advice rather than theory.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and B200.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on 4090, keeping MoE serving in mind from the start.

If someone wants the exact settings I can clean them up and post them.