prefill bottlenecks on longer prompts for Mistral

[bytestack]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.

[cozyarch]

following this because I'm in the same boat.

[rocmhouse90]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

If you post your exact settings, people here can probably narrow it down pretty fast.

[thetanest]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

following this because I'm in the same boat.