I'm trying to sanity-check my setup.

My current setup is vLLM with mini PC, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use. The logs were somehow calmer than I was.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on A6000, keeping small local stacks in mind from the start.

If someone wants the exact settings I can clean them up and post them.

Did a bit of testing and thought I'd post the result here.

After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.

Happy to post more details if anyone wants them.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and M3 Max felt surprisingly usable once I tuned around continuous batching.

If someone wants the exact settings I can clean them up and post them.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with reranking.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Qwen, vLLM, dual 3090, with a lot of attention on structured output.

Maybe useful as a starting point for newer folks. I keep promising myself this is the last rebuild.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and B200 felt surprisingly usable once I tuned around tokens/sec.

Maybe useful as a starting point for newer folks.

Did a bit of testing and thought I'd post the result here.

I tried this with DeepSeek using SGLang on B200. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

I figured I'd ask here before changing everything.

My current setup is LM Studio with A6000, and I keep running into NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?

Did a bit of testing and thought I'd post the result here.

After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Curious if anyone else landed in the same place.