memory layers vs plain long context

[mellowsupply]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on Threadripper box, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

[fastfield4]

Spent some time on this today and wanted to compare notes.

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

I'm happy to test one more variant if you want a second data point.