This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like long context decides everything downstream.

Curious whether that lines up with what other people are seeing. The logs were somehow calmer than I was.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate prefix caching until they actually try to run this stuff for more than one afternoon.

Maybe obvious to some of you, but it clicked for me this week.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about continuous batching.

I'm especially interested in mistakes worth avoiding.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and SGLang for DeepSeek. The interesting difference for me has been tokens/sec.

Interested in tradeoffs more than fan loyalty.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like structured output decides everything downstream.

Wondering if this is just where the whole space is heading now.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate KV cache pressure until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like small local stacks decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like long context decides everything downstream.

Wondering if this is just where the whole space is heading now.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

Short version: vLLM behaved well for me, but only after I stopped ignoring HBM constraints.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

The part that surprised me was how much the boring details mattered more than the model choice.