embedding drift issues for Llama

[cachelab]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of MLX and A6000 felt surprisingly usable once I tuned around embeddings.

Maybe useful as a starting point for newer folks.

[macropoint]

Appreciate the detail. I ended up in about the same place using vLLM.

[phaseport36]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

If you post your exact settings, people here can probably narrow it down pretty fast. At this point the real bottleneck is my patience.

[vramarch]

Good point. I ended up in about the same place using SGLang. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[dockbrook21]

Not sure I agree there. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Worth testing the simpler explanation before adopting the more interesting one.

[kernelmike]

following this because I'm in the same boat.

[deepseeked]

Thanks, that helps. I ended up in about the same place using Ollama.

[signaltrack44]

Thanks, that helps. That matches what I saw on B200. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.