do small edge boxes make sense

[cacheplane98]

I'm trying to sanity-check my setup.

My current setup is vLLM with mini PC, and I keep running into small local stacks.

Not looking for the perfect answer, mostly what worked in real use. The logs were somehow calmer than I was.

[turbosupply83]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Worth testing the simpler explanation before adopting the more interesting one.

[llmtrack]

Good point. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[signaltrack44]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

[sysstation69]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quali

worth checking the driver first.

[emberrunner]

following this because I'm in the same boat.

[batchpeak]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tail latency. Once you know the plain path works, then add the faster path back in piece by piece.

[routearch]

That matches what I was seeing. The pattern around reranking has been pretty similar for me. The biggest change for me was simplifying the stack instead of adding one more optimization layer.