Help with Search / Retrieval / Reranking

[forgemixer]

Curious what people here are seeing.

My current setup is MLX with GB200, and I keep running into prefill/decode split.

If you've tested something similar, what did you keep and what did you stop bothering with?

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[embershed99]

I don't think that's the whole story. I get why that seems right, but on B200 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.