+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=50)
+---- Thread: high-context local setup (/showthread.php?tid=1173)
high-context local setup - turboarch95 - 10-13-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like batch size decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
Re: high-context local setup - plainember52 - 10-13-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?