prefill bottlenecks on longer prompts: early impressions

[dockkite69]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around vLLM and 4090.

Simple and boring is a valid answer.

I care more about stability than squeezing the last possible tokens/sec out of it.

[deltagrid]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?