best sandbox for trying risky tools

[vramghost]

This keeps coming up in conversation so I figured I'd start a thread.

It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with INT4.

Maybe obvious to some of you, but it clicked for me this week.

I expected the biggest difference to come from the model, but the stack around it mattered at least as much.

[phasenode]

Tried to reproduce it just now. On B200 with MLX, I saw basically the same behavior.

[amberkite]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[edgekite]

I think the bottleneck is somewhere else. I get why that seems right, but on B200 the bottleneck felt elsewhere.

[cozyarch]

Thanks, that helps. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[localtinker]

Thanks, that helps. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[rocmpath]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it. I keep promising myself this is the last rebuild.

[mangohouse]

worth checking the driver first.

[audiodesk]

worth checking the driver first.

I'd start with the simpler stack.

I'm happy to test one more variant if you want a second data point.

[southden]

that fixed it for me too.