best local stack on mac right now with Ollama

[mlxgarden]

I'm trying to sanity-check my setup.

My current setup is PyTorch with dual 3090, and I keep running into HBM constraints.

If you've tested something similar, what did you keep and what did you stop bothering with? At this point the real bottleneck is my patience.

[mlxgarden]

I don't think that's the whole story. I've seen people blame INT4 when the real issue was much more mundane.

[visionyard23]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

I'm happy to test one more variant if you want a second data point.

[labnest]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[sanescope]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The surprise was that memory pressure showed up before throughput collapsed.

[turboyard94]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[torchstation]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

What worked for me was pretty simple. If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[cudacrate]

I hit almost the same issue last night.

[sanescope]

I hit almost the same issue last night.

I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[quiettoken]

I hit almost the same issue last night.

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.