prefill bottlenecks on longer prompts: anyone testing this?

[sanescope]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between llama.cpp and TensorRT-LLM for DeepSeek. The interesting difference for me has been batch size.

Interested in tradeoffs more than fan loyalty.

I care more about stability than squeezing the last possible tokens/sec out of it.