Embeddings discussion

[ggufwire22]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on GB200, or squeeze more out of the stack and deal with the complexity.

Interested in tradeoffs more than fan loyalty.

[torchdesk]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[edgecoast]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. I expected the model choice to matter more than it did.

[tinycontext]

Good point. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[torchdesk]

Not sure I agree there. I get why that seems right, but on 4090 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[paperport64]

What exact setup are you running? Is this local-only or are you routing anything externally?

[edgequill]

Good point. I ended up in about the same place using LM Studio. The biggest change for me was simplifying the stack instead of adding one more optimiz

Good point. That matches what I saw on A6000. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[vramscope]

Interesting, I hadn't tried that. I ended up in about the same place using MLX.