+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: do small edge boxes make sense (/showthread.php?tid=345)
do small edge boxes make sense - cacheplane98 - 09-19-2026
I'm trying to sanity-check my setup.
My current setup is vLLM with mini PC, and I keep running into small local stacks.
Not looking for the perfect answer, mostly what worked in real use. The logs were somehow calmer than I was.
Re: do small edge boxes make sense - turbosupply83 - 09-19-2026
Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.
Re: do small edge boxes make sense - llmtrack - 09-22-2026
Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: do small edge boxes make sense - signaltrack44 - 09-22-2026
Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.
Re: do small edge boxes make sense - sysstation69 - 09-22-2026
Quote:Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quali
worth checking the driver first.
Re: do small edge boxes make sense - emberrunner - 09-22-2026
following this because I'm in the same boat.
Re: do small edge boxes make sense - batchpeak - 09-22-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.
Re: do small edge boxes make sense - routearch - 09-23-2026
That matches what I was seeing. The pattern around reranking has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.