Help with Embeddings on 4090

[southtinker]

I figured I'd ask here before changing everything.

My current setup is TensorRT-LLM with 7900 XTX, and I keep running into NVFP4.

If you've tested something similar, what did you keep and what did you stop bothering with?

[vectorshed43]

Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.

[paperport64]

I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[audiorunner74]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[vectorshed43]

Can you fill in a bit more context? Is this local-only or are you routing anything externally?

[vramscope]

What exact setup are you running? What model, what runtime, and roughly what context size?

[deltapoint17]

What exact setup are you running? What model, what runtime, and roughly what context size?

same here on my side.

[quantnest]

What exact setup are you running? What model, what runtime, and roughly what context size?

following this because I'm in the same boat.

[edgecoast]

What exact setup are you running? What model, what runtime, and roughly what context size?

Not sure I agree there. I've seen people blame long context when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[torchdesk]

I hit almost the same issue last night.