do small edge boxes make sense

[relayforge]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Llama, llama.cpp, B200, with a lot of attention on continuous batching.

Maybe useful as a starting point for newer folks.

[localpatch78]

Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[pixelcrate82]

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[macrostack69]

Appreciate the detail. I ended up in about the same place using MLX. The biggest change for me was simplifying the stack instead of adding one more optimization layer. At this point the real bottleneck is my patience.

[mlxgarden]

Maybe, but I had the opposite result. I get why that seems right, but on A6000 the bottleneck felt elsewhere.

[cozywire]

I hit almost the same issue last night.

[kernelhouse70]

Need a little more detail here. Do you see the same thing with a smaller batch or shorter prompt?