+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: LLM Models (https://www.datasculpting.ai/inference/forumdisplay.php?fid=12)
+--- Forum: Open Weight (https://www.datasculpting.ai/inference/forumdisplay.php?fid=49)
+---- Forum: Hosted (https://www.datasculpting.ai/inference/forumdisplay.php?fid=50)
+---- Thread: best setup on 24GB VRAM for Phi (/showthread.php?tid=60)
best setup on 24GB VRAM for Phi - sparrowkite53 - 08-28-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like small local stacks decides everything downstream.
Wondering if this is just where the whole space is heading now. I keep promising myself this is the last rebuild.
Re: best setup on 24GB VRAM for Phi - thetashed24 - 08-28-2026
A couple things I'd check first. On B200, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.
Re: best setup on 24GB VRAM for Phi - tensorriver - 08-29-2026
What worked for me was pretty simple. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives.
Re: best setup on 24GB VRAM for Phi - torchsmith25 - 08-29-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher.
Re: best setup on 24GB VRAM for Phi - sparrowshed - 08-29-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at small local stacks.
Re: best setup on 24GB VRAM for Phi - modbench - 08-29-2026
following this because I'm in the same boat.
Re: best setup on 24GB VRAM for Phi - vectorfield - 08-31-2026
Thanks, that helps. I ended up in about the same place using SGLang. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: best setup on 24GB VRAM for Phi - thetarunner - 08-31-2026
A couple things I'd check first. On 7900 XTX, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.