+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Miscellaneous Discussion (https://www.datasculpting.ai/inference/forumdisplay.php?fid=24)
+--- Forum: Off Topic (https://www.datasculpting.ai/inference/forumdisplay.php?fid=27)
+--- Thread: desk setup thread (/showthread.php?tid=1366)
Pages:
1
2
desk setup thread - phaseshed - 10-16-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.
Trying to avoid rebuilding the whole thing twice. The logs were somehow calmer than I was.
Re: desk setup thread - rocmcrate26 - 10-17-2026
Quote:Spent some time on this today and wanted to compare notes.
that fixed it for me too.
Re: desk setup thread - vectorport - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: desk setup thread - phaseshed - 10-18-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.
Re: desk setup thread - stackpilot - 10-18-2026
Did a small comparison this afternoon. On Threadripper box with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: desk setup thread - phaseport36 - 10-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.
Re: desk setup thread - quietstack - 10-18-2026
I think the bottleneck is somewhere else. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: desk setup thread - cudascope - 10-19-2026
Quote:Did a small comparison this afternoon. On Threadripper box with PyTorch, I saw basically the same behavior. I expected the model choice to matter more
Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.
Re: desk setup thread - sparrowkite53 - 10-19-2026
worth checking the driver first.
Re: desk setup thread - quiettoken - 10-19-2026
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.