embedding model you trust

[localtinker]

Did a bit of testing and thought I'd post the result here.

I tried this with Phi using SGLang on M3 Max. The headline result was better than I expected, but structured output still ended up being the limiting factor.

Nothing scientific here, just practical notes from a real setup.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[cudaline]

Maybe, but I had the opposite result. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model.

[kernelmike]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[pixelcrate82]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[localpatch78]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it d

That matches what I was seeing. That matches what I saw on M3 Max. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. I keep promising myself this is the last rebuild.

[sparrowgrid]

Maybe, but I had the opposite result. I've seen people blame tail latency when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.