Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, MLX, M3 Max, with a lot of attention on batch size.

Maybe useful as a starting point for newer folks.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, LM Studio, M3 Max, with a lot of attention on long context.

Maybe useful as a starting point for newer folks.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between SGLang and llama.cpp for Mixtral. The interesting difference for me has been long context.

What would you pick if you had to live with it for a few months?

Did a bit of testing and thought I'd post the result here.

After a few runs on M3 Max, I ended up with decent performance, but the setup only really made sense once I adjusted around tool calling.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once long context becomes important.

Trying to avoid rebuilding the whole thing twice.

Maybe I'm missing something obvious.

I'm mostly working with Qwen in Ollama on A6000. The part I'm trying to understand is token cost.

Not looking for the perfect answer, mostly what worked in real use.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

I care more about stability than squeezing the last possible tokens/sec out of it.

Did a bit of testing and thought I'd post the result here.

After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.

I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

Would appreciate practical advice rather than theory.