+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Training & Customization (https://www.datasculpting.ai/inference/forumdisplay.php?fid=13)
+--- Forum: Dataset Creation / Curation (https://www.datasculpting.ai/inference/forumdisplay.php?fid=56)
+--- Thread: reranking still worth it for small sets (/showthread.php?tid=406)
Pages:
1
2
reranking still worth it for small sets - audiopoint18 - 09-21-2026
I figured I'd ask here before changing everything.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.
If you've tested something similar, what did you keep and what did you stop bothering with?
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: reranking still worth it for small sets - audioforge - 09-22-2026
Quote:I figured I'd ask here before changing everything.
I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: reranking still worth it for small sets - relaysupply90 - 09-22-2026
That matches what I was seeing. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: reranking still worth it for small sets - sparrowhall - 09-23-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output.
Re: reranking still worth it for small sets - llm_wren - 09-23-2026
I hit almost the same issue last night.
Re: reranking still worth it for small sets - sparrowghost - 09-23-2026
Tried to reproduce it just now. On M3 Max with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did.
I'm happy to test one more variant if you want a second data point.
Re: reranking still worth it for small sets - torchsupply99 - 09-26-2026
What exact setup are you running? Are you testing this on mini PC or something else?
Re: reranking still worth it for small sets - llm_wren - 09-26-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: reranking still worth it for small sets - phasespark - 09-27-2026
that fixed it for me too.
Re: reranking still worth it for small sets - signalcraft14 - 09-27-2026
A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.