+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context for Llama (/showthread.php?tid=27)
Pages:
1
2
working config for 32k context for Llama - mlxgarden - 08-24-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on dual 3090, I ended up with decent performance, but the setup only really made sense once I adjusted around prefill/decode split.
Curious if anyone else landed in the same place.
Re: working config for 32k context for Llama - rocmriley - 08-25-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: working config for 32k context for Llama - mangoquill - 08-25-2026
Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
Appreciate the detail. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: working config for 32k context for Llama - rocmhouse90 - 08-25-2026
Quote:Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
Appreciate the detail. The pattern around batch size has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: working config for 32k context for Llama - vectorport - 08-25-2026
I hit almost the same issue last night. At this point the real bottleneck is my patience.
Re: working config for 32k context for Llama - rocmhouse90 - 08-25-2026
Quote:I hit almost the same issue last night. At this point the real bottleneck is my patience.
I'd approach it in a boring order. On M3 Max, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: working config for 32k context for Llama - audioriver - 08-26-2026
I hit almost the same issue last night.
Re: working config for 32k context for Llama - rocmcrate26 - 08-28-2026
Quote:I hit almost the same issue last night.
Thanks, that helps. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: working config for 32k context for Llama - rocmhouse90 - 08-28-2026
worth checking the driver first.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: working config for 32k context for Llama - rasterbench - 08-30-2026
Appreciate the detail. That matches what I saw on 7900 XTX.