containment for model-driven automation: worth it?

[fusedriver]

Did a bit of testing and thought I'd post the result here.

I tried this with Qwen using Ollama on 7900 XTX. The headline result was better than I expected, but quant drift still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

[vramscope]

Did a bit of testing and thought I'd post the result here.

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals.

[sparrowgrid]

Good point. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[torchshed66]

Good point. I ended up in about the same place using MLX. Worth rechecking your defaults though, because one bad assumption can make the rest of the t

Interesting, I hadn't tried that. The pattern around small local stacks has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[routearch]

Ran a quick check on my box. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[forgemixer]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[llmtrack26]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[cudapeak]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[northgrid33]

Not sure I agree there. I've seen people blame INT4 when the real issue was much more mundane.

[southspark]

subscribed, curious where this lands.

I think the bottleneck is somewhere else. I've seen people blame embeddings when the real issue was much more mundane.