RAG chunking that actually helped on Threadripper box

[tensorriver]

I think the bottleneck is somewhere else. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[mangopath4]

That matches what I was seeing. I don't think your conclusion is far off. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[byteyard]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there.

[phaseport36]

A couple questions before I guess too hard. What model, what runtime, and roughly what context size?

[thetaplane]

Not sure I agree there. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. The logs were somehow calmer than I was.

[orbitline]

Ran a quick check on my box. My result was mixed: first run looked bad, second run looked normal. The surprise was that memory pressure showed up before throughput collapsed.