Maybe I'm missing something obvious.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with?

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

I care more about stability than squeezing the last possible tokens/sec out of it.

I'm trying to sanity-check my setup.

My current setup is PyTorch with 7900 XTX, and I keep running into consumer GPUs.

Would appreciate practical advice rather than theory.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate small local stacks until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

I care more about stability than squeezing the last possible tokens/sec out of it.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of SGLang and mini PC felt surprisingly usable once I tuned around tokens/sec.

Feel free to tear it apart if you see something dumb.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with batch size.

Wondering if this is just where the whole space is heading now.

Did a bit of testing and thought I'd post the result here.

I tried this with Qwen using TensorRT-LLM on A6000. The headline result was better than I expected, but consumer GPUs still ended up being the limiting factor.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about consumer GPUs.

I'm especially interested in mistakes worth avoiding.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between LM Studio and llama.cpp for Llama. The interesting difference for me has been reranking.

Interested in tradeoffs more than fan loyalty.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.