Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on 5090, keeping tool calling in mind from the start.

Maybe useful as a starting point for newer folks.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and SGLang for Llama. The interesting difference for me has been tokens/sec.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like token cost decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This is mostly for local work rather than public serving.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on dual 3090, keeping NVFP4 in mind from the start.

Maybe useful as a starting point for newer folks.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, Ollama, Threadripper box, with a lot of attention on quant drift.

Feel free to tear it apart if you see something dumb.

Has anyone else run into this?

My current setup is LM Studio with M3 Max, and I keep running into continuous batching.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Did a bit of testing and thought I'd post the result here.

Short version: Ollama behaved well for me, but only after I stopped ignoring embeddings.

Nothing scientific here, just practical notes from a real setup.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between TensorRT-LLM and Ollama for Mixtral. The interesting difference for me has been INT4.

What would you pick if you had to live with it for a few months?

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.