This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like continuous batching decides everything downstream.

Curious whether that lines up with what other people are seeing.

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using vLLM on B200. The headline result was better than I expected, but workstation thermals still ended up being the limiting factor.

Happy to post more details if anyone wants them.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate token cost until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

The part that surprised me was how much the boring details mattered more than the model choice. At this point the real bottleneck is my patience.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, vLLM, mini PC, with a lot of attention on consumer GPUs.

Feel free to tear it apart if you see something dumb.

Curious what people here are seeing.

I'm mostly working with DeepSeek in MLX on GB200. The part I'm trying to understand is prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefill/decode split until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with workstation thermals.

Curious whether that lines up with what other people are seeing.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, Ollama, 7900 XTX, with a lot of attention on prefix caching.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

I figured I'd ask here before changing everything.

I'm mostly working with FLUX in Ollama on Threadripper box. The part I'm trying to understand is long context.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and 5090 felt surprisingly usable once I tuned around token cost.

Feel free to tear it apart if you see something dumb.