+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Hardware (https://www.datasculpting.ai/inference/forumdisplay.php?fid=4)
+--- Forum: CPUs / NPUs / Accelerators (https://www.datasculpting.ai/inference/forumdisplay.php?fid=30)
+--- Thread: do small edge boxes make sense (/showthread.php?tid=692)
do small edge boxes make sense - relayforge - 10-03-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on continuous batching.
Maybe useful as a starting point for newer folks.
Re: do small edge boxes make sense - localpatch78 - 10-03-2026
Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: do small edge boxes make sense - pixelcrate82 - 10-06-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: do small edge boxes make sense - macrostack69 - 10-07-2026
Appreciate the detail. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer. At this point the real bottleneck is my patience.
Re: do small edge boxes make sense - mlxgarden - 10-08-2026
Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.
Re: do small edge boxes make sense - cozywire - 10-08-2026
I hit almost the same issue last night.
Re: do small edge boxes make sense - kernelhouse70 - 10-09-2026
Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?