+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=50)
+---- Thread: high-context local setup for Llama (/showthread.php?tid=7)
Pages:
1
2
high-context local setup for Llama - sparrowkite53 - 08-22-2026
Spent some time on this today and wanted to compare notes.
I'm comparing two approaches here: keep it simple on 4090, or squeeze more out of the stack and deal with the complexity.
What would you pick if you had to live with it for a few months?
Re: high-context local setup for Llama - dockriver - 08-22-2026
I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
Re: high-context local setup for Llama - modrunner27 - 08-22-2026
Quote:I tested this tonight. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at quant drift. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: high-context local setup for Llama - quiettoken - 08-22-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at reranking. Once you know the plain path works, then add the faster path back in piece by piece.
Re: high-context local setup for Llama - rocmriley - 08-22-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: high-context local setup for Llama - cobaltlab - 08-23-2026
following this because I'm in the same boat.
Re: high-context local setup for Llama - vramarch - 08-23-2026
Need a little more detail here. Are you testing this on M3 Max or something else?
Re: high-context local setup for Llama - audiodesk - 08-23-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: high-context local setup for Llama - modrunner27 - 08-26-2026
that fixed it for me too.
Re: high-context local setup for Llama - cacheline - 08-26-2026
Quote:that fixed it for me too.
What exact setup are you running? Are you testing this on Threadripper box or something else?