I figured I'd ask here before changing everything.

My current setup is PyTorch with M3 Max, and I keep running into router policy.

Not looking for the perfect answer, mostly what worked in real use.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and SGLang for Mixtral. The interesting difference for me has been tail latency.

Trying to avoid rebuilding the whole thing twice.

Maybe I'm missing something obvious.

I'm mostly working with FLUX in SGLang on A6000. The part I'm trying to understand is KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with? The logs were somehow calmer than I was.

Has anyone else run into this?

My current setup is Ollama with M3 Max, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate tool calling until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on 5090, keeping tool calling in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around consumer GPUs.

Curious if anyone else landed in the same place.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: TensorRT-LLM on Threadripper box, keeping HBM constraints in mind from the start.

Maybe useful as a starting point for newer folks.