I'm trying to sanity-check my setup.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling reranking.

Not looking for the perfect answer, mostly what worked in real use.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like batch size decides everything downstream.

Curious whether that lines up with what other people are seeing.

Did a bit of testing and thought I'd post the result here.

I tried this with Llama using vLLM on GB200. The headline result was better than I expected, but KV cache pressure still ended up being the limiting factor.

Curious if anyone else landed in the same place.

This is mostly for local work rather than public serving.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once MoE serving becomes important.

What would you pick if you had to live with it for a few months?

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: TensorRT-LLM on B200, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

This is mostly for local work rather than public serving.

Curious what people here are seeing.

I'm mostly working with Kimi in TensorRT-LLM on 7900 XTX. The part I'm trying to understand is workstation thermals.

Would appreciate practical advice rather than theory.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on 4090, keeping NVFP4 in mind from the start.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate small local stacks until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Would rather learn from your scars than collect my own all at once.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once tail latency becomes important.

Interested in tradeoffs more than fan loyalty.

This is mostly for local work rather than public serving.