RAG chunking that actually helped: worth it?

[dockbrook21]

Posting this in case it saves someone else a little time.

I'm helping a small team set up a local workflow. We don't need a giant stack, but we do care about long context.

Simple and boring is a valid answer.

This is mostly for local work rather than public serving.

[fastfield4]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous advice, but repeatable baselines save a lot of time. Turns out 'just one more test' was a lie.

[localpatch]

A couple things I'd check first. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at MoE serving. Not glamorous

I think the bottleneck is somewhere else. I've seen people blame workstation thermals when the real issue was much more mundane. Could be a workload difference, but I wouldn't generalize too fast from one setup.

[audioforge]

Appreciate the detail. That matches what I saw on 4090. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[quietcraft]

Interesting, I hadn't tried that. I don't think your conclusion is far off. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[bytesupply76]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.

[mintstation99]

What worked for me was pretty simple. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamo

That matches what I was seeing. I ended up in about the same place using llama.cpp. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. I keep promising myself this is the last rebuild.

[localpatch]

Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing.

[llmtrack]

Need a little more detail here. Are you testing this on 7900 XTX or something else?

[macrostack69]

What exact setup are you running? Are you testing this on GB200 or something else?