I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling small local stacks.

Would appreciate practical advice rather than theory.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once continuous batching becomes important.

Trying to avoid rebuilding the whole thing twice.

I'm trying to sanity-check my setup.

I'm mostly working with Llama in LM Studio on A6000. The part I'm trying to understand is tail latency.

Not looking for the perfect answer, mostly what worked in real use.

I figured I'd ask here before changing everything.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 7900 XTX.

Would rather learn from your scars than collect my own all at once.

Did a bit of testing and thought I'd post the result here.

I tried this with Kimi using MLX on A6000. The headline result was better than I expected, but batch size still ended up being the limiting factor.

Happy to post more details if anyone wants them.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between Ollama and llama.cpp for Llama. The interesting difference for me has been tail latency.

What would you pick if you had to live with it for a few months?

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, Ollama, 4090, with a lot of attention on quant drift.

Maybe useful as a starting point for newer folks.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on B200, keeping tokens/sec in mind from the start.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.