+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Inference Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=9)
+--- Forum: Other Engines (https://www.datasculpting.ai/inference/forumdisplay.php?fid=39)
+--- Thread: prefill bottlenecks on longer prompts on Threadripper box (/showthread.php?tid=818)
prefill bottlenecks on longer prompts on Threadripper box - vramarch - 10-06-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once VRAM headroom becomes important.
What would you pick if you had to live with it for a few months?
Re: prefill bottlenecks on longer prompts on Threadripper box - quiethouse - 10-08-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: prefill bottlenecks on longer prompts on Threadripper box - mintstation99 - 10-08-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: prefill bottlenecks on longer prompts on Threadripper box - mintquill - 10-09-2026
I hit almost the same issue last night.
Re: prefill bottlenecks on longer prompts on Threadripper box - vrampilot - 10-10-2026
Interesting, I hadn't tried that. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: prefill bottlenecks on longer prompts on Threadripper box - phasescope - 10-10-2026
Quote:Interesting, I hadn't tried that. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just loo
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.
Re: prefill bottlenecks on longer prompts on Threadripper box - rasterbench - 10-10-2026
A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.