+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: prefill bottlenecks on longer prompts for Mistral (/showthread.php?tid=289)
Pages:
1
2
prefill bottlenecks on longer prompts for Mistral - phaseshed - 09-16-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Maybe useful as a starting point: MLX on 5090, keeping prefill/decode split in mind from the start.
Maybe useful as a starting point for newer folks.
Re: prefill bottlenecks on longer prompts for Mistral - fastscope - 09-17-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
worth checking the driver first.
Re: prefill bottlenecks on longer prompts for Mistral - vramport - 09-17-2026
Quote:Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
That matches what I was seeing. That matches what I saw on GB200. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prefill bottlenecks on longer prompts for Mistral - kernelghost - 09-20-2026
My first guess would be this: If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: prefill bottlenecks on longer prompts for Mistral - rocmcrate26 - 09-20-2026
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: prefill bottlenecks on longer prompts for Mistral - batchcraft - 09-20-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you know the plain path works, then add the faster path back in piece by piece.
Re: prefill bottlenecks on longer prompts for Mistral - fastfield4 - 09-20-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Once you
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: prefill bottlenecks on longer prompts for Mistral - quantghost - 09-21-2026
Interesting, I hadn't tried that. The pattern around tokens/sec has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
I'm happy to test one more variant if you want a second data point. Turns out 'just one more test' was a lie.
Re: prefill bottlenecks on longer prompts for Mistral - visionyard23 - 09-21-2026
Thanks, that helps. The pattern around embeddings has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
I'm happy to test one more variant if you want a second data point.
Re: prefill bottlenecks on longer prompts for Mistral - vramridge - 09-24-2026
Interesting, I hadn't tried that. The pattern around structured output has been pretty similar for me.