10-18-2026, 08:04 PM
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
throughput matters, but tail latency matters too
|
router in front of vLLM: early impressions
|
|
« Next Oldest | Next Newest »
|
Users browsing this thread: 1 Guest(s)