+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Search / Retrieval / Reranking (https://www.datasculpting.ai/inference/forumdisplay.php?fid=61)
+--- Thread: Search / Retrieval / Reranking discussion on A6000 (/showthread.php?tid=333)
Pages:
1
2
Search / Retrieval / Reranking discussion on A6000 - macropoint - 09-18-2026
I'm trying to sanity-check my setup.
My current setup is vLLM with M3 Max, and I keep running into KV cache pressure.
Not looking for the perfect answer, mostly what worked in real use.
Re: Search / Retrieval / Reranking discussion on A6000 - routeforge64 - 09-18-2026
Can you fill in a bit more context? What model, what runtime, and roughly what context size?
Re: Search / Retrieval / Reranking discussion on A6000 - tensorriver - 09-19-2026
I'd start with the simpler stack.
Re: Search / Retrieval / Reranking discussion on A6000 - mintstation99 - 09-19-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Mistral and compare from there. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Search / Retrieval / Reranking discussion on A6000 - latentscope35 - 09-19-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
Re: Search / Retrieval / Reranking discussion on A6000 - macrotinker - 09-19-2026
Quote:Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.
that fixed it for me too.
Re: Search / Retrieval / Reranking discussion on A6000 - phasestream53 - 09-20-2026
Maybe, but I had the opposite result. I've seen people blame speculative decoding when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: Search / Retrieval / Reranking discussion on A6000 - signaltrack44 - 09-20-2026
Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: Search / Retrieval / Reranking discussion on A6000 - rocmriley - 09-20-2026
Quote:Good point. I ended up in about the same place using Ollama. Worth rechecking your defaults though, because one bad assumption can make the rest of th
Thanks, that helps. That matches what I saw on A6000. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.
Re: Search / Retrieval / Reranking discussion on A6000 - localcraft17 - 09-20-2026
That matches what I was seeing. The pattern around long context has been pretty similar for me.