Embeddings discussion for Phi

[rocmmixer]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of TensorRT-LLM and A6000 felt surprisingly usable once I tuned around MoE serving.

Maybe useful as a starting point for newer folks.

[dockriver]

I hit almost the same issue last night.

[nanopatch]

that fixed it for me too.

[quantnest]

Interesting, I hadn't tried that. That matches what I saw on mini PC. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[oxidemixer]

same here on my side.

[paperport64]

same here on my side.

Thanks, that helps. The pattern around router policy has been pretty similar for me. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[torchsmith25]

Interesting, I hadn't tried that. I ended up in about the same place using LM Studio.

[paperport64]

A couple things I'd check first. On 4090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.