+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Dataset Creation / Curation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=56)
+--- Thread: reranking still worth it for small sets (/showthread.php?tid=26)
Pages:
1
2
reranking still worth it for small sets - rocmcrate26 - 08-24-2026
Has anyone else run into this?
I'm mostly working with DeepSeek in LM Studio on GB200. The part I'm trying to understand is KV cache pressure.
If you've tested something similar, what did you keep and what did you stop bothering with?
This is mostly for local work rather than public serving.
Re: reranking still worth it for small sets - emberpeak - 08-26-2026
I'd start with the simpler stack.
Re: reranking still worth it for small sets - mintquill - 08-26-2026
same here on my side.
Re: reranking still worth it for small sets - llm_wren - 08-26-2026
I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: reranking still worth it for small sets - rocmstack25 - 08-26-2026
Quote:I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once
following this because I'm in the same boat.
Re: reranking still worth it for small sets - cozyarch - 08-26-2026
Quote:[quote]I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obviou
My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: reranking still worth it for small sets - amberkite - 08-27-2026
Appreciate the detail. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: reranking still worth it for small sets - fastscope - 08-27-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: reranking still worth it for small sets - northgarden37 - 08-27-2026
Quote:What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.
Re: reranking still worth it for small sets - amberhall - 08-27-2026
A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.