Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: vLLM on GB200, keeping KV cache pressure in mind from the start.

If someone wants the exact settings I can clean them up and post them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it. The logs were somehow calmer than I was.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: DeepSeek, TensorRT-LLM, 7900 XTX, with a lot of attention on KV cache pressure.

Feel free to tear it apart if you see something dumb.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Has anyone else run into this?

I'm mostly working with DeepSeek in PyTorch on dual 3090. The part I'm trying to understand is consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and 4090 felt surprisingly usable once I tuned around structured output.

Feel free to tear it apart if you see something dumb.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Mistral, vLLM, A6000, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of SGLang and A6000 felt surprisingly usable once I tuned around quant drift.

Maybe useful as a starting point for newer folks.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.