Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and llama.cpp for Mistral. The interesting difference for me has been MoE serving.

What would you pick if you had to live with it for a few months?

Has anyone else run into this?

My current setup is Ollama with Threadripper box, and I keep running into structured output.

Would appreciate practical advice rather than theory.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I figured I'd ask here before changing everything.

My current setup is llama.cpp with A6000, and I keep running into tool calling.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefill/decode split decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

Maybe I'm missing something obvious.

I'm mostly working with Gemma in SGLang on M3 Max. The part I'm trying to understand is prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about small local stacks.

I'm especially interested in mistakes worth avoiding.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on Threadripper box, keeping speculative decoding in mind from the start.

If someone wants the exact settings I can clean them up and post them.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, MLX, dual 3090, with a lot of attention on structured output.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with small local stacks.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.