why did throughput drop after update

[thetaplane]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between vLLM and vLLM for Kimi. The interesting difference for me has been reranking.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[promptpoint]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[phasestream53]

Did a small comparison this afternoon. On A6000 with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.