reranking still worth it for small sets

[audiopoint18]

I figured I'd ask here before changing everything.

I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling speculative decoding.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[audioforge]

I figured I'd ask here before changing everything.

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[relaysupply90]

That matches what I was seeing. That matches what I saw on 4090. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[sparrowhall]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at structured output.

[llm_wren]

I hit almost the same issue last night.

[sparrowghost]

Tried to reproduce it just now. On M3 Max with LM Studio, I saw basically the same behavior. I expected the model choice to matter more than it did.

I'm happy to test one more variant if you want a second data point.

[torchsupply99]

What exact setup are you running? Are you testing this on mini PC or something else?

[llm_wren]

What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[phasespark]

that fixed it for me too.

[signalcraft14]

A couple things I'd check first. On Threadripper box, the easiest win is often to back off the aggressive settings and see whether the problem survives. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.