Wanted to compare notes with people doing similar things.

I'm mostly working with Phi in Ollama on A6000. The part I'm trying to understand is continuous batching.

Not looking for the perfect answer, mostly what worked in real use.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, Ollama, A6000, with a lot of attention on embeddings.

Feel free to tear it apart if you see something dumb.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and LM Studio for Qwen. The interesting difference for me has been speculative decoding.

Trying to avoid rebuilding the whole thing twice.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and TensorRT-LLM for Llama. The interesting difference for me has been tokens/sec.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

The part that surprised me was how much the boring details mattered more than the model choice.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

Would rather learn from your scars than collect my own all at once.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

The part that surprised me was how much the boring details mattered more than the model choice.

Did a bit of testing and thought I'd post the result here.

Short version: llama.cpp behaved well for me, but only after I stopped ignoring batch size.

Happy to post more details if anyone wants them.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.