what truly stays local on B200

[rasterbench]

Spent some time on this today and wanted to compare notes.

Two paths both seem viable, but they break down differently once workstation thermals becomes important.

Trying to avoid rebuilding the whole thing twice.

[mlxgarden]

I'd start with the simpler stack. The logs were somehow calmer than I was.

[deltagrid]

Good point. The pattern around tool calling has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[mlxgarden]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[audioriver]

Ran a quick check on my box. On A6000 with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.

[rasterbench]

A couple questions before I guess too hard. Are you testing this on 4090 or something else?

[rocmcrate26]

Appreciate the detail. That matches what I saw on 5090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[deltagrid]

That matches what I was seeing. That matches what I saw on Threadripper box. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[flintriver50]

Can you fill in a bit more context? Are you testing this on 5090 or something else?

[quantshed]

My first guess would be this: If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious.