This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate reranking until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and LM Studio for Gemma. The interesting difference for me has been router policy.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around NVFP4.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once speculative decoding becomes important.

Trying to avoid rebuilding the whole thing twice.

This is mostly for local work rather than public serving.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and Threadripper box felt surprisingly usable once I tuned around tokens/sec.

Feel free to tear it apart if you see something dumb.

I care more about stability than squeezing the last possible tokens/sec out of it.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like INT4 decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

This is mostly for local work rather than public serving.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and SGLang for Llama. The interesting difference for me has been NVFP4.

Trying to avoid rebuilding the whole thing twice.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.