+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Embeddings (https://www.datasculpting.ai/inference/forumdisplay.php?fid=59)
+--- Thread: Help with Embeddings on B200 (/showthread.php?tid=759)
Help with Embeddings on B200 - ggufwire22 - 10-04-2026
This keeps coming up in conversation so I figured I'd start a thread.
It feels like the conversation has shifted hard toward inference and serving details lately, and honestly that makes sense once you start dealing with tool calling.
Wondering if this is just where the whole space is heading now.
This is mostly for local work rather than public serving.
Re: Help with Embeddings on B200 - torchdesk - 10-04-2026
A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?
Re: Help with Embeddings on B200 - vectorshed43 - 10-04-2026
That matches what I was seeing. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.
Re: Help with Embeddings on B200 - paperport64 - 10-05-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: Help with Embeddings on B200 - vramscope - 10-05-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there.
Re: Help with Embeddings on B200 - vectorshed43 - 10-06-2026
What exact setup are you running? Do you see the same thing with a smaller batch or shorter prompt?
Re: Help with Embeddings on B200 - rasterhall - 10-06-2026
A couple questions before I guess too hard. What model, what runtime, and roughly what context size?
Re: Help with Embeddings on B200 - edgecoast - 10-06-2026
What worked for me was pretty simple. Before rebuilding anything, I'd test one clean baseline with DeepSeek and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
I'm happy to test one more variant if you want a second data point.
Re: Help with Embeddings on B200 - rasterhall - 10-06-2026
Did a small comparison this afternoon. On 4090 with MLX, I saw basically the same behavior. The main thing that moved was latency, not quality.