high-context local setup for Llama

[sparrowkite53]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

[dockriver]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[modrunner27]

I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[quiettoken]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmriley]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[cobaltlab]

following this because I'm in the same boat.

[vramarch]

Need a little more detail here. Are you testing this on M3 Max or something else?

[audiodesk]

What exact setup are you running? What model, what runtime, and roughly what context size?

If you post your exact settings, people here can probably narrow it down pretty fast.

[modrunner27]

that fixed it for me too.

[cacheline]

that fixed it for me too.

What exact setup are you running? Are you testing this on Threadripper box or something else?