Spent some time on this today and wanted to compare notes.

I've gone back and forth between SGLang and PyTorch for Kimi. The interesting difference for me has been continuous batching.

Trying to avoid rebuilding the whole thing twice.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like tail latency decides everything downstream.

Curious whether that lines up with what other people are seeing.

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tokens/sec.

If you've tested something similar, what did you keep and what did you stop bothering with?

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on 5090, keeping long context in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring NVFP4.

Curious if anyone else landed in the same place.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once small local stacks becomes important.

Trying to avoid rebuilding the whole thing twice.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around router policy.

Feel free to tear it apart if you see something dumb.

I'm trying to sanity-check my setup.

My current setup is vLLM with M3 Max, and I keep running into KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.

Simple and boring is a valid answer.