prefill bottlenecks on longer prompts on Threadripper box

[vramarch]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once VRAM headroom becomes important.

What would you pick if you had to live with it for a few months?

[quiethouse]

What exact setup are you running? Is this local-only or are you routing anything externally?

[mintstation99]

I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[mintquill]

I hit almost the same issue last night.

[vrampilot]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[phasescope]

Interesting, I hadn't tried that. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just loo

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[rasterbench]

A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives.