reranking still worth it for small sets

[phasenode]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on A6000, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

[plainscope50]

Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.

If you post your exact settings, people here can probably narrow it down pretty fast.

[labnest]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.

[deepseeked]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal.

[mlxgarden]

Appreciate the detail. I don't think your conclusion is far off.

[batchfield55]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at workstation thermals. Once you know the plain path works, then add the faster path back in piece by piece.