prefill bottlenecks on longer prompts on A6000

[batchscope72]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Gemma, MLX, 4090, with a lot of attention on MoE serving.

If someone wants the exact settings I can clean them up and post them.

[tensorworks]

Thought I'd post a small recipe because a lot of threads ask similar things.

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[deepseeked]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.

[tinycontext]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[modrunner27]

Thanks, that helps. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[turboscope]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[routearch]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[homelabhenry]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[benchplease]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the

Appreciate the detail. I ended up in about the same place using vLLM.

[byteyard]

I tested this tonight. On GB200 with Ollama, I saw basically the same behavior. Turns out 'just one more test' was a lie.