+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Embeddings (https://www.datasculpting.ai/inference/forumdisplay.php?fid=59)
+--- Thread: Embeddings discussion (/showthread.php?tid=436)
Pages:
1
2
Embeddings discussion - quantnest - 09-23-2026
Curious what people here are seeing.
I'm trying to decide whether to keep the current stack or change course. Main question is how people here are handling tail latency.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: Embeddings discussion - ggufwire22 - 09-23-2026
What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: Embeddings discussion - vramscope - 09-23-2026
Quote:What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A lot of
Thanks, that helps. I don't think your conclusion is far off.
Re: Embeddings discussion - edgecoast - 09-23-2026
Quote:[quote]What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. A
Thanks, that helps. That matches what I saw on Threadripper box. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
I'm happy to test one more variant if you want a second data point.
Re: Embeddings discussion - vectorshed43 - 09-23-2026
Did a small comparison this afternoon. I couldn't reproduce the slowdown until I pushed context much higher. The surprise was that memory pressure showed up before throughput collapsed.
Re: Embeddings discussion - paperport64 - 09-24-2026
That matches what I was seeing. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Embeddings discussion - vramscope - 09-25-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: Embeddings discussion - torchdesk - 09-25-2026
Did a small comparison this afternoon. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.
Re: Embeddings discussion - quantnest - 09-25-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: Embeddings discussion - ggufwire22 - 09-25-2026
I'd approach it in a boring order. If you're on PyTorch, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.