Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about embeddings.

Would rather learn from your scars than collect my own all at once.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate token cost until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using PyTorch on B200. The headline result was better than I expected, but speculative decoding still ended up being the limiting factor.

Curious if anyone else landed in the same place.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

Would rather learn from your scars than collect my own all at once.

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around INT4.

Curious if anyone else landed in the same place.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with token cost.

Wondering if this is just where the whole space is heading now.

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around reranking.

Curious if anyone else landed in the same place.

Did a bit of testing and thought I'd post the result here.

I tried this with Gemma using PyTorch on 5090. The headline result was better than I expected, but embeddings still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Kimi, Ollama, B200, with a lot of attention on quant drift.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.