This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like quant drift decides everything downstream.

Wondering if this is just where the whole space is heading now.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once router policy becomes important.

Trying to avoid rebuilding the whole thing twice.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mixtral, PyTorch, mini PC, with a lot of attention on token cost.

Maybe useful as a starting point for newer folks.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once batch size becomes important.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.

Curious what people here are seeing.

My current setup is PyTorch with 5090, and I keep running into tool calling.

Would appreciate practical advice rather than theory.

I care more about stability than squeezing the last possible tokens/sec out of it.

Maybe I'm missing something obvious.

My current setup is llama.cpp with GB200, and I keep running into prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around VRAM headroom.

Nothing scientific here, just practical notes from a real setup.

I care more about stability than squeezing the last possible tokens/sec out of it.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and MLX for DeepSeek. The interesting difference for me has been long context.

Interested in tradeoffs more than fan loyalty.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and dual 3090.

I'm especially interested in mistakes worth avoiding.