small win thread

[quiettoken]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once prefill/decode split becomes important.

Trying to avoid rebuilding the whole thing twice.

[plaingarden18]

Interesting, I hadn't tried that. That matches what I saw on M3 Max.

[quiettoken]

Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[plaingarden18]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[cacheplane98]

Need a little more detail here. Are you testing this on 4090 or something else?

[quiettoken]

What exact setup are you running? Is this local-only or are you routing anything externally?

[ggufwire22]

My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[thetagrid15]

A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?

[ggufwire22]

A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece. At this point the real bottleneck is my patience.

[marbletrack]

same here on my side.

I'm happy to test one more variant if you want a second data point.