Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and Ollama for FLUX. The interesting difference for me has been KV cache pressure.

Interested in tradeoffs more than fan loyalty.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once embeddings becomes important.

Trying to avoid rebuilding the whole thing twice.

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like NVFP4 decides everything downstream.

Wondering if this is just where the whole space is heading now.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around llama.cpp and 4090.

I'm especially interested in mistakes worth avoiding.

This is mostly for local work rather than public serving.

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate reranking until they actually try to run this stuff for more than one afternoon.

Curious whether that lines up with what other people are seeing.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around MLX and 7900 XTX.

Simple and boring is a valid answer.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

Did a bit of testing and thought I'd post the result here.

After a few runs on 7900 XTX, I ended up with decent performance, but the setup only really made sense once I adjusted around MoE serving.

Happy to post more details if anyone wants them.

Did a bit of testing and thought I'd post the result here.

After a few runs on GB200, I ended up with decent performance, but the setup only really made sense once I adjusted around workstation thermals.

Curious if anyone else landed in the same place.

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once workstation thermals becomes important.

What would you pick if you had to live with it for a few months?

This is mostly for local work rather than public serving.

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around PyTorch and 5090.

Simple and boring is a valid answer.