Has anyone else run into this?

My current setup is LM Studio with dual 3090, and I keep running into consumer GPUs.

If you've tested something similar, what did you keep and what did you stop bothering with?

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 7900 XTX, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and 4090 felt surprisingly usable once I tuned around tail latency.

Feel free to tear it apart if you see something dumb.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Kimi, TensorRT-LLM, dual 3090, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs. I keep promising myself this is the last rebuild.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: PyTorch on Threadripper box, keeping prefix caching in mind from the start.

Maybe useful as a starting point for newer folks.

I care more about stability than squeezing the last possible tokens/sec out of it. At this point the real bottleneck is my patience.

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with continuous batching.

Maybe obvious to some of you, but it clicked for me this week.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, dual 3090, with a lot of attention on HBM constraints.

Maybe useful as a starting point for newer folks.

I figured I'd ask here before changing everything.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling MoE serving.

Would appreciate practical advice rather than theory.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate tail latency until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once tail latency becomes important.

Interested in tradeoffs more than fan loyalty.