Did a bit of testing and thought I'd post the result here.

After a few runs on B200, I ended up with decent performance, but the setup only really made sense once I adjusted around speculative decoding.

Curious if anyone else landed in the same place.

I care more about stability than squeezing the last possible tokens/sec out of it.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate batch size until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Interested in tradeoffs more than fan loyalty.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and TensorRT-LLM for Mistral. The interesting difference for me has been long context.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on M3 Max, keeping reranking in mind from the start.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

Maybe I'm missing something obvious.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling router policy.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.

Did a bit of testing and thought I'd post the result here.

After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Wanted to compare notes with people doing similar things.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling structured output.

Would appreciate practical advice rather than theory. The logs were somehow calmer than I was.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: FLUX, MLX, 4090, with a lot of attention on router policy.

Maybe useful as a starting point for newer folks. I keep promising myself this is the last rebuild.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like embeddings decides everything downstream.

Wondering if this is just where the whole space is heading now.