Did a bit of testing and thought I'd post the result here.

Short version: vLLM behaved well for me, but only after I stopped ignoring VRAM headroom.

Curious if anyone else landed in the same place.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and A6000 felt surprisingly usable once I tuned around speculative decoding.

If someone wants the exact settings I can clean them up and post them.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, Ollama, 4090, with a lot of attention on small local stacks.

Maybe useful as a starting point for newer folks.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and B200 felt surprisingly usable once I tuned around prefill/decode split.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling embeddings.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and M3 Max.

Simple and boring is a valid answer.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on dual 3090, keeping tail latency in mind from the start.

If someone wants the exact settings I can clean them up and post them.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of PyTorch and M3 Max felt surprisingly usable once I tuned around KV cache pressure.

If someone wants the exact settings I can clean them up and post them.