Search / Retrieval / Reranking discussion with PyTorch

[phasespark]

Appreciate the detail. I ended up in about the same place using llama.cpp. I got better results once I stopped chasing one metric and just looked at w

Ran a quick check on my box. On M3 Max with TensorRT-LLM, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[batchsize]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[llmarch83]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at HBM constraints. A lot of the

I hit almost the same issue last night.

[mangoquill]

Good point. I ended up in about the same place using MLX. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[cudascope]

Interesting, I hadn't tried that. The pattern around reranking has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.