+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: RAG & Knowledge Systems (https://www.datasculpting.ai/inference/forumdisplay.php?fid=14)
+--- Forum: Embeddings (https://www.datasculpting.ai/inference/forumdisplay.php?fid=59)
+--- Thread: Help with Embeddings on 4090 (/showthread.php?tid=600)
Pages:
1
2
Help with Embeddings on 4090 - southtinker - 09-30-2026
I figured I'd ask here before changing everything.
My current setup is TensorRT-LLM with 7900 XTX, and I keep running into NVFP4.
If you've tested something similar, what did you keep and what did you stop bothering with?
Re: Help with Embeddings on 4090 - vectorshed43 - 09-30-2026
Ran a quick check on my box. I couldn't reproduce the slowdown until I pushed context much higher. I expected the model choice to matter more than it did.
Re: Help with Embeddings on 4090 - paperport64 - 10-01-2026
I'd approach it in a boring order. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.
Re: Help with Embeddings on 4090 - audiorunner74 - 10-01-2026
A couple questions before I guess too hard. Is this local-only or are you routing anything externally?
Re: Help with Embeddings on 4090 - vectorshed43 - 10-01-2026
Can you fill in a bit more context? Is this local-only or are you routing anything externally?
Re: Help with Embeddings on 4090 - vramscope - 10-02-2026
What exact setup are you running? What model, what runtime, and roughly what context size?
Re: Help with Embeddings on 4090 - deltapoint17 - 10-03-2026
Quote:What exact setup are you running? What model, what runtime, and roughly what context size?
same here on my side.
Re: Help with Embeddings on 4090 - quantnest - 10-04-2026
Quote:Quote:What exact setup are you running? What model, what runtime, and roughly what context size?
following this because I'm in the same boat.
Re: Help with Embeddings on 4090 - edgecoast - 10-04-2026
Quote:Quote:Quote:What exact setup are you running? What model, what runtime, and roughly what context size?
Not sure I agree there. I've seen people blame long context when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.
Re: Help with Embeddings on 4090 - torchdesk - 10-04-2026
I hit almost the same issue last night.