where reranking mattered: worth it?

[tinycontext]

Posting this in case it saves someone else a little time.

I'm not trying to build a perfect architecture diagram. I just want a sane starting point around Ollama and A6000.

Would rather learn from your scars than collect my own all at once.

If I had to summarize the lesson so far, it would be that clean setup and observability beat clever guesses.

[homelabhenry]

That matches what I was seeing. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[modbench]

That matches what I was seeing. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can

worth checking the driver first.

I'm happy to test one more variant if you want a second data point.

[homelabhenry]

Good point. The pattern around router policy has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[tensorfield88]

Thanks, that helps. The pattern around consumer GPUs has been pretty similar for me.

[rocmmixer]

Thanks, that helps. The pattern around consumer GPUs has been pretty similar for me.

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[homelabhenry]

What exact setup are you running? What model, what runtime, and roughly what context size?

[labpath]

Good point. I ended up in about the same place using llama.cpp.

[homelabhenry]

That matches what I was seeing. I ended up in about the same place using TensorRT-LLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[sanepeak48]

Did a small comparison this afternoon. On 4090 with Ollama, I saw basically the same behavior. I expected the model choice to matter more than it did.