Help with Embeddings: what are you seeing?

[edgecoast]

Did a bit of testing and thought I'd post the result here.

I tried this with DeepSeek using SGLang on 7900 XTX. The headline result was better than I expected, but VRAM headroom still ended up being the limiting factor.

Curious if anyone else landed in the same place.

The part that surprised me was how much the boring details mattered more than the model choice.

[vectorshed43]

A couple things I'd check first. On mini PC, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[paperport64]

worth checking the driver first.

[llmtrack26]

Thanks, that helps. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[vramscope]

Thanks, that helps. That matches what I saw on 4090.

[edgecoast]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.

I'm happy to test one more variant if you want a second data point.

[paperport64]

Tried to reproduce it just now. On 5090 with LM Studio, I saw basically the same behavior. The main thing that moved was latency, not quality.

[kernelhouse70]

same here on my side.

[torchdesk]

What worked for me was pretty simple. If you're on SGLang, I'd confirm the version first and then reduce variables until the behavior is obvious. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[edgecoast]

I think the bottleneck is somewhere else. I get why that seems right, but on 7900 XTX the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.