09-05-2026, 09:39 PM
Quote:[quote][quote]My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressu
Interesting, I hadn't tried that. I ended up in about the same place using vLLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.