desk setup thread

[phaseshed]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on M3 Max, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice. The logs were somehow calmer than I was.

[rocmcrate26]

Spent some time on this today and wanted to compare notes.

that fixed it for me too.

[vectorport]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at prefix caching. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[phaseshed]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[stackpilot]

Did a small comparison this afternoon. On Threadripper box with PyTorch, I saw basically the same behavior. I expected the model choice to matter more than it did.

[phaseport36]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at router policy. Not glamorous advice, but repeatable baselines save a lot of time.

[quietstack]

I think the bottleneck is somewhere else. I've seen people blame INT4 when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[cudascope]

Did a small comparison this afternoon. On Threadripper box with PyTorch, I saw basically the same behavior. I expected the model choice to matter more

Not sure I agree there. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Worth testing the simpler explanation before adopting the more interesting one.

[sparrowkite53]

worth checking the driver first.

[quiettoken]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Phi and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.