Has anyone else run into this?

I'm mostly working with DeepSeek in llama.cpp on Threadripper box. The part I'm trying to understand is batch size.

Not looking for the perfect answer, mostly what worked in real use.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and dual 3090 felt surprisingly usable once I tuned around prefix caching.

If someone wants the exact settings I can clean them up and post them.

The part that surprised me was how much the boring details mattered more than the model choice.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like consumer GPUs decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once tool calling becomes important.

Interested in tradeoffs more than fan loyalty.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on B200, keeping continuous batching in mind from the start.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

I'm trying to sanity-check my setup.

I'm mostly working with Mixtral in PyTorch on mini PC. The part I'm trying to understand is MoE serving.

If you've tested something similar, what did you keep and what did you stop bothering with?

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about KV cache pressure.

Would rather learn from your scars than collect my own all at once.

I care more about stability than squeezing the last possible tokens/sec out of it.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.

Simple and boring is a valid answer.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using TensorRT-LLM on 5090. The headline result was better than I expected, but HBM constraints still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on 4090, keeping VRAM headroom in mind from the start.

Maybe useful as a starting point for newer folks.