prefill bottlenecks on longer prompts

[oxidemixer]

My first guess would be this: If you're on vLLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.