desktop apps around local AI on 5090

[vectorport]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between PyTorch and vLLM for FLUX. The interesting difference for me has been consumer GPUs.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.

[llmpeak]

What exact setup are you running? Is this local-only or are you routing anything externally?

[kernelrunner]

What exact setup are you running? Is this local-only or are you routing anything externally?

I'd approach it in a boring order. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[mangoquill]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[deltagrid]

Did a small comparison this afternoon. On 4090 with vLLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[audionode8]

Thanks, that helps. That matches what I saw on M3 Max.

[batchfield55]

that fixed it for me too.

[llmpeak]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost.

[cachebench]

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorport]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly.