Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of LM Studio and GB200 felt surprisingly usable once I tuned around continuous batching.

Maybe useful as a starting point for newer folks.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

I figured I'd ask here before changing everything.

My current setup is SGLang with dual 3090, and I keep running into workstation thermals.

Not looking for the perfect answer, mostly what worked in real use.

I care more about stability than squeezing the last possible tokens/sec out of it.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and B200.

I'm especially interested in mistakes worth avoiding.

Wanted to compare notes with people doing similar things.

My current setup is PyTorch with 5090, and I keep running into long context.

Would appreciate practical advice rather than theory.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and llama.cpp for Mistral. The interesting difference for me has been quant drift.

What would you pick if you had to live with it for a few months?

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Did a bit of testing and thought I'd post the result here.

I tried this with Mixtral using llama.cpp on 4090. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.

Curious if anyone else landed in the same place. Turns out 'just one more test' was a lie.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once token cost becomes important.

Trying to avoid rebuilding the whole thing twice.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.