+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: General (https://www.datasculpting.ai/inference/forumdisplay.php?fid=25)
+--- Thread: small win thread (/showthread.php?tid=609)
small win thread - quiettoken - 10-01-2026
Spent some time on this today and wanted to compare notes.
Two paths both seem viable, but they break down differently once prefill/decode split becomes important.
Trying to avoid rebuilding the whole thing twice.
Re: small win thread - plaingarden18 - 10-01-2026
Interesting, I hadn't tried that. That matches what I saw on M3 Max.
Re: small win thread - quiettoken - 10-02-2026
Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: small win thread - plaingarden18 - 10-03-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: small win thread - cacheplane98 - 10-03-2026
Need a little more detail here. Are you testing this on 4090 or something else?
Re: small win thread - quiettoken - 10-03-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: small win thread - ggufwire22 - 10-06-2026
My first guess would be this: On dual 3090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.
Re: small win thread - thetagrid15 - 10-09-2026
A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?
Re: small win thread - ggufwire22 - 10-09-2026
Quote:A couple questions before I guess too hard. Are you testing this on Threadripper box or something else?
My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece. At this point the real bottleneck is my patience.
Re: small win thread - marbletrack - 10-09-2026
same here on my side.
I'm happy to test one more variant if you want a second data point.