RTX 5090 local inference

[turbogrid11]

Did a bit of testing and thought I'd post the result here.

Short version: vLLM behaved well for me, but only after I stopped ignoring NVFP4.

Happy to post more details if anyone wants them.

[cudapoint]

Did a bit of testing and thought I'd post the result here.

Appreciate the detail. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[embersmith]

That matches what I was seeing. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[cudapoint]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[batchwire]

Appreciate the detail. The pattern around speculative decoding has been pretty similar for me.

[routeforge64]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.

[signaltrack44]

subscribed, curious where this lands.

[nanobrook]

I'd approach it in a boring order. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time. The logs were somehow calmer than I was.

[rocmriley]

I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[paperrunner]

following this because I'm in the same boat.