Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around quant drift.

Happy to post more details if anyone wants them.

I care more about stability than squeezing the last possible tokens/sec out of it.

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling token cost.

Would appreciate practical advice rather than theory.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with KV cache pressure.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.

Did a bit of testing and thought I'd post the result here.

I tried this with FLUX using MLX on A6000. The headline result was better than I expected, but tail latency still ended up being the limiting factor.

Curious if anyone else landed in the same place.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, vLLM, 7900 XTX, with a lot of attention on small local stacks.

Feel free to tear it apart if you see something dumb.

I'm trying to sanity-check my setup.

My current setup is SGLang with A6000, and I keep running into long context.

If you've tested something similar, what did you keep and what did you stop bothering with?

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it. I keep promising myself this is the last rebuild.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, 5090, with a lot of attention on prefill/decode split.

Feel free to tear it apart if you see something dumb.

Curious what people here are seeing.

My current setup is TensorRT-LLM with M3 Max, and I keep running into KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.