reranking still worth it for small sets: setup advice

[mellowstream]

I figured I'd ask here before changing everything.

My current setup is llama.cpp with A6000, and I keep running into tool calling.

If you've tested something similar, what did you keep and what did you stop bothering with?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[relaysupply90]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[mangopeak72]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious.

[edgekite]

That matches what I was seeing. The pattern around long context has been pretty similar for me.

[benchplease]

Good point. I don't think your conclusion is far off.

[northquill]

Need a little more detail here. What model, what runtime, and roughly what context size?

[localcraft17]

Appreciate the detail. I ended up in about the same place using llama.cpp.

[quantghost]

Appreciate the detail. I ended up in about the same place using llama.cpp.

Not sure I agree there. I get why that seems right, but on 5090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[thetapath]

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there.