Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: Ollama on GB200, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, TensorRT-LLM, M3 Max, with a lot of attention on tail latency.

Feel free to tear it apart if you see something dumb.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: TensorRT-LLM on mini PC, keeping consumer GPUs in mind from the start.

Maybe useful as a starting point for newer folks.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like long context decides everything downstream.

Curious whether that lines up with what other people are seeing.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once MoE serving becomes important.

What would you pick if you had to live with it for a few months?

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with NVFP4.

Maybe obvious to some of you, but it clicked for me this week.

Curious what people here are seeing.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling prefill/decode split.

Not looking for the perfect answer, mostly what worked in real use.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate structured output until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

Curious what people here are seeing.

I'm mostly working with Kimi in PyTorch on B200. The part I'm trying to understand is INT4.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once router policy becomes important.

Trying to avoid rebuilding the whole thing twice.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.