+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Vector Databases (https://www.datasculpting.ai/inference/forumdisplay.php?fid=60)
+--- Thread: Vector Databases discussion (/showthread.php?tid=276)
Vector Databases discussion - macroforge - 09-15-2026
Did a bit of testing and thought I'd post the result here.
I tried this with Gemma using PyTorch on 5090. The headline result was better than I expected, but embeddings still ended up being the limiting factor.
Nothing scientific here, just practical notes from a real setup.
The part that surprised me was how much the boring details mattered more than the model choice.
Re: Vector Databases discussion - routeforge64 - 09-15-2026
I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.
Re: Vector Databases discussion - tinycontext - 09-15-2026
What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
I'm happy to test one more variant if you want a second data point.
Re: Vector Databases discussion - orbitbench6 - 09-15-2026
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. Once you know the plain path works, then add the faster path back in piece by piece.
Re: Vector Databases discussion - emberquill - 09-16-2026
same here on my side.
Re: Vector Databases discussion - macronest - 09-16-2026
Can you fill in a bit more context? Do you see the same thing with a smaller batch or shorter prompt?
Re: Vector Databases discussion - tinycontext - 09-17-2026
My first guess would be this: If you're on TensorRT-LLM, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Vector Databases discussion - relayforge - 09-17-2026
Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal.
Re: Vector Databases discussion - vectorshed43 - 09-18-2026
subscribed, curious where this lands.
Re: Vector Databases discussion - byteyard - 09-18-2026
Good point. I ended up in about the same place using llama.cpp.