Apple unified memory limits on mini PC

[torchshed]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like prefix caching decides everything downstream.

Curious whether that lines up with what other people are seeing.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[thetaarch43]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[mellowstream]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?

If you post your exact settings, people here can probably narrow it down pretty fast.

[plainember52]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[labnest]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was l

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mixtral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[mellowstream]

[quote]Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that move

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[plainember52]

My first guess would be this: On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.