+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: prefill bottlenecks on longer prompts on A6000 (/showthread.php?tid=916)
Pages:
1
2
prefill bottlenecks on longer prompts on A6000 - batchscope72 - 10-08-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Gemma, MLX, 4090, with a lot of attention on MoE serving.
If someone wants the exact settings I can clean them up and post them.
Re: prefill bottlenecks on longer prompts on A6000 - tensorworks - 10-08-2026
Quote:Thought I'd post a small recipe because a lot of threads ask similar things.
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: prefill bottlenecks on longer prompts on A6000 - deepseeked - 10-08-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.
Re: prefill bottlenecks on longer prompts on A6000 - tinycontext - 10-08-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: prefill bottlenecks on longer prompts on A6000 - modrunner27 - 10-08-2026
Thanks, that helps. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prefill bottlenecks on longer prompts on A6000 - turboscope - 10-09-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: prefill bottlenecks on longer prompts on A6000 - routearch - 10-10-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: prefill bottlenecks on longer prompts on A6000 - homelabhenry - 10-11-2026
I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: prefill bottlenecks on longer prompts on A6000 - benchplease - 10-11-2026
Quote:I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the
Appreciate the detail. I ended up in about the same place using vLLM.
Re: prefill bottlenecks on longer prompts on A6000 - byteyard - 10-11-2026
I tested this tonight. On GB200 with Ollama, I saw basically the same behavior. Turns out 'just one more test' was a lie.