Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like INT4 decides everything downstream.

Curious whether that lines up with what other people are seeing.

This is mostly for local work rather than public serving.

I'm trying to sanity-check my setup.

My current setup is Ollama with GB200, and I keep running into continuous batching.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

Has anyone else run into this?

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling structured output.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Wondering if this is just where the whole space is heading now.

The part that surprised me was how much the boring details mattered more than the model choice.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on 5090, keeping quant drift in mind from the start.

Maybe useful as a starting point for newer folks.

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around tokens/sec.

Curious if anyone else landed in the same place.

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

Would appreciate practical advice rather than theory.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and vLLM for Gemma. The interesting difference for me has been prefill/decode split.

Interested in tradeoffs more than fan loyalty.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 7900 XTX, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.