+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Embeddings (https://www.datasculpting.ai/inference/forumdisplay.php?fid=59)
+--- Thread: Help with Embeddings (/showthread.php?tid=522)
Pages:
1
2
Help with Embeddings - torchdesk - 09-27-2026
Did a bit of testing and thought I'd post the result here.
After a few runs on mini PC, I ended up with decent performance, but the setup only really made sense once I adjusted around tail latency.
Nothing scientific here, just practical notes from a real setup.
This is mostly for local work rather than public serving.
Re: Help with Embeddings - vectorshed43 - 09-29-2026
Not sure I agree there. I've seen people blame speculative decoding when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.
Re: Help with Embeddings - vramscope - 09-29-2026
What exact setup are you running? Is this local-only or are you routing anything externally?
Re: Help with Embeddings - rasterhall - 09-30-2026
Quote:What exact setup are you running? Is this local-only or are you routing anything externally?
Tried to reproduce it just now. On A6000 with MLX, I saw basically the same behavior. I expected the model choice to matter more than it did.
Re: Help with Embeddings - vramscope - 10-01-2026
following this because I'm in the same boat.
Re: Help with Embeddings - edgecoast - 10-01-2026
Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
If you post your exact settings, people here can probably narrow it down pretty fast.
Re: Help with Embeddings - rasterhall - 10-04-2026
Thanks, that helps. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: Help with Embeddings - quantnest - 10-04-2026
My first guess would be this: If you're on LM Studio, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: Help with Embeddings - ggufwire22 - 10-05-2026
Thanks, that helps. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Help with Embeddings - paperport64 - 10-05-2026
I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher. The main thing that moved was latency, not quality.