prefill bottlenecks on longer prompts on A6000

[mintstream9]

Appreciate the detail. The pattern around MoE serving has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mlxgarden]

I don't think that's the whole story. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere.

[tinycontext]

I don't think that's the whole story. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere.

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

I'm happy to test one more variant if you want a second data point. Turns out 'just one more test' was a lie.