where reranking mattered on mini PC

[bytelab]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[tinycontext]

Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. I expected the model choice to matter more than it did.

[homelabhenry]

that fixed it for me too.

[ggufvault]

Tried to reproduce it just now. On GB200 with llama.cpp, I saw basically the same behavior. I expected the model choice to matter more than it did.

[labpath]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[embershed99]

I don't think that's the whole story. I've seen people blame prefill/decode split when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup. Turns out 'just one more test' was a lie.