reranking still worth it for small sets

[rocmcrate26]

Has anyone else run into this?

I'm mostly working with DeepSeek in LM Studio on GB200. The part I'm trying to understand is KV cache pressure.

If you've tested something similar, what did you keep and what did you stop bothering with?

This is mostly for local work rather than public serving.

[emberpeak]

I'd start with the simpler stack.

[mintquill]

same here on my side.

[llm_wren]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.

[rocmstack25]

I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Once

following this because I'm in the same boat.

[cozyarch]

[quote]I'd approach it in a boring order. If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obviou

My first guess would be this: Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[amberkite]

Appreciate the detail. I ended up in about the same place using TensorRT-LLM. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[fastscope]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

[northgarden37]

What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at tool calling. Not glamorous advice, but repeatable baselines save a lot of time.

[amberhall]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at KV cache pressure. Once you know the plain path works, then add the faster path back in piece by piece.