Did a bit of testing and thought I'd post the result here.

Short version: llama.cpp behaved well for me, but only after I stopped ignoring quant drift.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of SGLang and B200 felt surprisingly usable once I tuned around NVFP4.

If someone wants the exact settings I can clean them up and post them.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between SGLang and PyTorch for FLUX. The interesting difference for me has been NVFP4.

Trying to avoid rebuilding the whole thing twice.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on GB200, keeping batch size in mind from the start.

Maybe useful as a starting point for newer folks.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

Would rather learn from your scars than collect my own all at once.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate reranking until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like quant drift decides everything downstream.

Curious whether that lines up with what other people are seeing.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of vLLM and Threadripper box felt surprisingly usable once I tuned around NVFP4.

Feel free to tear it apart if you see something dumb.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and Threadripper box felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.