+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: do small edge boxes make sense with vLLM (/showthread.php?tid=32)
do small edge boxes make sense with vLLM - audionode8 - 08-26-2026
This keeps coming up in conversation so I figured I'd start a thread.
The more I work with local setups, the more it seems like VRAM headroom decides everything downstream.
Maybe obvious to some of you, but it clicked for me this week.
Re: do small edge boxes make sense with vLLM - routecrate - 08-26-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Phi and compare from there.