This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

I care more about stability than squeezing the last possible tokens/sec out of it.

Wanted to compare notes with people doing similar things.

I'm mostly working with Kimi in Ollama on Threadripper box. The part I'm trying to understand is long context.

If you've tested something similar, what did you keep and what did you stop bothering with?

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and MLX for Gemma. The interesting difference for me has been HBM constraints.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

I figured I'd ask here before changing everything.

I'm mostly working with Mixtral in vLLM on M3 Max. The part I'm trying to understand is NVFP4.

Not looking for the perfect answer, mostly what worked in real use.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefix caching.

I'm especially interested in mistakes worth avoiding.

Did a bit of testing and thought I'd post the result here.

Short version: PyTorch behaved well for me, but only after I stopped ignoring quant drift.

Curious if anyone else landed in the same place.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on 4090, keeping long context in mind from the start.

Maybe useful as a starting point for newer folks.

Did a bit of testing and thought I'd post the result here.

Short version: SGLang behaved well for me, but only after I stopped ignoring NVFP4.

Curious if anyone else landed in the same place.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate INT4 until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.