prefill/decode separation in practice

[turboarch95]

Thanks, that helps. The pattern around reranking has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cacheline]

Thanks, that helps. The pattern around reranking has been pretty similar for me. I got better results once I stopped chasing one metric and just looke

following this because I'm in the same boat.

[oxidemixer]

[quote]Thanks, that helps. The pattern around reranking has been pretty similar for me. I got better results once I stopped chasing one metric and jus

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at long context. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[turboarch95]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?