Help with Search / Retrieval / Reranking on GB200

[vectorpoint]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: MLX on A6000, keeping small local stacks in mind from the start.

If someone wants the exact settings I can clean them up and post them.

[oxidestream]

Ran a quick check on my box. On dual 3090 with vLLM, I saw basically the same behavior.

[dockriver]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?