Search / Retrieval / Reranking discussion on A6000

[macropoint]

I'm trying to sanity-check my setup.

My current setup is vLLM with M3 Max, and I keep running into KV cache pressure.

Not looking for the perfect answer, mostly what worked in real use.

[routeforge64]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[tensorriver]

I'd start with the simpler stack.

[mintstation99]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.

[latentscope35]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[macrotinker]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

that fixed it for me too.

[phasestream53]

Maybe, but I had the opposite result. I've seen people blame speculative decoding when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[signaltrack44]

Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[rocmriley]

Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of th

Thanks, that helps. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[localcraft17]

That matches what I was seeing. The pattern around long context has been pretty similar for me.