+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=50)
+---- Thread: high-context local setup: anyone testing this? (/showthread.php?tid=602)
Pages:
1
2
Re: high-context local setup: anyone testing this? - thetarunner - 10-04-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: high-context local setup: anyone testing this? - cozyarch - 10-04-2026
Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: high-context local setup: anyone testing this? - sparrowshed - 10-05-2026
worth checking the driver first.
Re: high-context local setup: anyone testing this? - northgrid33 - 10-05-2026
Good point. I ended up in about the same place using vLLM.
Re: high-context local setup: anyone testing this? - cozyarch - 10-05-2026
I'd start with the simpler stack.
Re: high-context local setup: anyone testing this? - emberrunner - 10-05-2026
Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.