industry move that matters for local users for Llama

[llmpeak]

Appreciate the detail. I ended up in about the same place using MLX.

[phaseshed]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at speculative decoding.

[bytestack]

that fixed it for me too.

[modbench]

Tried to reproduce it just now. On M3 Max with Ollama, I saw basically the same behavior. I expected the model choice to matter more than it did.

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[modbench]

Not sure I agree there. I've seen people blame batch size when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

I'm happy to test one more variant if you want a second data point.

[southpatch]

Interesting, I hadn't tried that. I ended up in about the same place using PyTorch. The biggest change for me was simplifying the stack instead of adding one more optimization layer.