vector DB that stayed out of the way

[quantyard]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about structured output.

Would rather learn from your scars than collect my own all at once.

[bytestack]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there.

[torchdesk]

Maybe, but I had the opposite result. I get why that seems right, but on mini PC the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.

[benchplease]

that fixed it for me too.

[vramarch]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not quality.

[sparrowshed]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly. The main thing that moved was latency, not qual

Interesting, I hadn't tried that. That matches what I saw on A6000.

[ggufvault]

That matches what I was seeing. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[northcraft90]

Interesting, I hadn't tried that. That matches what I saw on mini PC. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[bytehall]

same here on my side.

[mangopeak72]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings.