I figured I'd ask here before changing everything.

My current setup is Ollama with B200, and I keep running into consumer GPUs.

Would appreciate practical advice rather than theory.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like batch size decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 5090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like NVFP4 decides everything downstream.

Wondering if this is just where the whole space is heading now.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like router policy decides everything downstream.

Curious whether that lines up with what other people are seeing.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around TensorRT-LLM and mini PC.

Would rather learn from your scars than collect my own all at once.

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Kimi, llama.cpp, A6000, with a lot of attention on router policy.

Feel free to tear it apart if you see something dumb.

The part that surprised me was how much the boring details mattered more than the model choice.

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using LM Studio on A6000. The headline result was better than I expected, but INT4 still ended up being the limiting factor.

Happy to post more details if anyone wants them.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once batch size becomes important.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.