Vector Databases discussion: anyone testing this?

[turboyard94]

Posting this in case it saves someone else a little time.

If you were starting from scratch today for this use case, what would you choose first and what would you ignore?

I'm especially interested in mistakes worth avoiding.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[deltagrid]

Good point. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[torchshed]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Kimi and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[llmarch51]

following this because I'm in the same boat.

[emberquill]

That matches what I was seeing. That matches what I saw on A6000.

[vramarch]

I don't think that's the whole story. I've seen people blame HBM constraints when the real issue was much more mundane. Worth testing the simpler explanation before adopting the more interesting one.

[fastfield4]

I'd approach it in a boring order. On A6000, the easiest win is often to back off the aggressive settings and see whether the problem survives. Once you know the plain path works, then add the faster path back in piece by piece.

[tinycontext]

Can you fill in a bit more context? Are you testing this on M3 Max or something else?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[orbitbench6]

Can you fill in a bit more context? Are you testing this on M3 Max or something else?

I hit almost the same issue last night.

[southspark]

Maybe, but I had the opposite result. I get why that seems right, but on 5090 the bottleneck felt elsewhere.