+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Local (https://www.datasculpting.ai/inference/forumdisplay.php?fid=51)
+---- Thread: working config for 32k context (/showthread.php?tid=1367)
working config for 32k context - cachebench - 10-17-2026
Posting this in case it saves someone else a little time.
I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about INT4.
Simple and boring is a valid answer.
For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.
Re: working config for 32k context - rocmriley - 10-18-2026
I think the bottleneck is somewhere else. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: working config for 32k context - audionode8 - 10-18-2026
same here on my side.
Re: working config for 32k context - llmpeak - 10-18-2026
I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at consumer GPUs.
Re: working config for 32k context - mangoquill - 10-18-2026
same here on my side.
Re: working config for 32k context - audioforge - 10-18-2026
Not sure I agree there. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.
Re: working config for 32k context - cacheplane98 - 10-19-2026
subscribed, curious where this lands.
Re: working config for 32k context - mangoquill - 10-19-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: working config for 32k context - mintstation99 - 10-19-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: working config for 32k context - modgarden - 10-19-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up
I'd start with the simpler stack.