high-context local setup

[turboarch95]

This keeps coming up in conversation so I figured I'd start a thread.

The more I work with local setups, the more it seems like batch size decides everything downstream.

Maybe obvious to some of you, but it clicked for me this week.

[plainember52]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?