Vector Databases discussion on M3 Max

[emberquill]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on B200, or squeeze more out of the stack and deal with the complexity.

Trying to avoid rebuilding the whole thing twice.

[localtinker]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[batchfield55]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at VRAM headroom. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

I'm happy to test one more variant if you want a second data point.

[fastfield4]

subscribed, curious where this lands.

If you post your exact settings, people here can probably narrow it down pretty fast.

[minthall]

Did a small comparison this afternoon. On 4090 with llama.cpp, I saw basically the same behavior.

[byteyard]

I think the bottleneck is somewhere else. I get why that seems right, but on dual 3090 the bottleneck felt elsewhere.

[amberkite]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.