Did a bit of testing and thought I'd post the result here.

I tried this with FLUX using TensorRT-LLM on dual 3090. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Small forum update from the Data Sculpting side.

We're still tuning the structure and getting everything in place. Feedback is welcome, especially if a topic feels like it has no obvious home yet.

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around batch size.

Curious if anyone else landed in the same place. The logs were somehow calmer than I was.

Maybe I'm missing something obvious.

I'm mostly working with Phi in llama.cpp on GB200. The part I'm trying to understand is NVFP4.

Would appreciate practical advice rather than theory.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about tokens/sec.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: LM Studio on mini PC, keeping continuous batching in mind from the start.

Feel free to tear it apart if you see something dumb.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and A6000.

Simple and boring is a valid answer.

The part that surprised me was how much the boring details mattered more than the model choice.

Maybe I'm missing something obvious.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling quant drift.

If you've tested something similar, what did you keep and what did you stop bothering with?

I care more about stability than squeezing the last possible tokens/sec out of it.

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on dual 3090, keeping VRAM headroom in mind from the start.

If someone wants the exact settings I can clean them up and post them.

I care more about stability than squeezing the last possible tokens/sec out of it.

Did a bit of testing and thought I'd post the result here.

After a few runs on Threadripper box, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Nothing scientific here, just practical notes from a real setup.

The part that surprised me was how much the boring details mattered more than the model choice.