that fixed it for me too. At this point the real bottleneck is my patience.

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Qwen, TensorRT-LLM, A6000, with a lot of attention on batch size.

If someone wants the exact settings I can clean them up and post them.

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and TensorRT-LLM for FLUX. The interesting difference for me has been tail latency.

Interested in tradeoffs more than fan loyalty.

Quote:subscribed, curious where this lands.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.

Appreciate the detail. I ended up in about the same place using Ollama. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

Good point. I ended up in about the same place using PyTorch. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about reranking.

I'm especially interested in mistakes worth avoiding.

I care more about stability than squeezing the last possible tokens/sec out of it.