Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

Has anyone else run into this?

I'm mostly working with Mistral in LM Studio on dual 3090. The part I'm trying to understand is VRAM headroom.

Would appreciate practical advice rather than theory.

Did a bit of testing and thought I'd post the result here.

I tried this with Mistral using MLX on M3 Max. The headline result was better than I expected, but continuous batching still ended up being the limiting factor.

Curious if anyone else landed in the same place.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate continuous batching until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on dual 3090, keeping prefill/decode split in mind from the start.

Maybe useful as a starting point for newer folks.

Has anyone else run into this?

My current setup is SGLang with 4090, and I keep running into prefix caching.

If you've tested something similar, what did you keep and what did you stop bothering with?

Wanted to compare notes with people doing similar things.

My current setup is LM Studio with dual 3090, and I keep running into HBM constraints.

Not looking for the perfect answer, mostly what worked in real use.

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

Small forum update from the Data Sculpting side.

We're still tuning the structure and getting everything in place. Feedback is welcome, especially if a topic feels like it has no obvious home yet.