Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about MoE serving.

Would rather learn from your scars than collect my own all at once.

Did a bit of testing and thought I'd post the result here.

Short version: LM Studio behaved well for me, but only after I stopped ignoring workstation thermals.

Happy to post more details if anyone wants them.

Did a bit of testing and thought I'd post the result here.

Short version: TensorRT-LLM behaved well for me, but only after I stopped ignoring tail latency.

Curious if anyone else landed in the same place.

Did a bit of testing and thought I'd post the result here.

After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Happy to post more details if anyone wants them.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about prefill/decode split.

Would rather learn from your scars than collect my own all at once.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and TensorRT-LLM for FLUX. The interesting difference for me has been long context.

Trying to avoid rebuilding the whole thing twice.

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and B200 felt surprisingly usable once I tuned around prefix caching.

Feel free to tear it apart if you see something dumb.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about speculative decoding.

Would rather learn from your scars than collect my own all at once.

Did a bit of testing and thought I'd post the result here.

After a few runs on 5090, I ended up with decent performance, but the setup only really made sense once I adjusted around long context.

Happy to post more details if anyone wants them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around HBM constraints.

Happy to post more details if anyone wants them.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.