prefill bottlenecks on longer prompts for Mistral

[phaseshed]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on 5090, keeping prefill/decode split in mind from the start.

Maybe useful as a starting point for newer folks.

[fastscope]

Thought I'd post a small recipe because a lot of threads ask similar things.

worth checking the driver first.

[vramport]

Thought I'd post a small recipe because a lot of threads ask similar things.

That matches what I was seeing. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[kernelghost]

My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[rocmcrate26]

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[batchcraft]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know the plain path works, then add the faster path back in piece by piece.

[fastfield4]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

If you post your exact settings, people here can probably narrow it down pretty fast.

[quantghost]

Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point. Turns out 'just one more test' was a lie.

[visionyard23]

Thanks, that helps. The pattern around embeddings has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

I'm happy to test one more variant if you want a second data point.

[vramridge]

Interesting, I hadn't tried that. The pattern around structured output has been pretty similar for me.