RAG chunking that actually helped on Threadripper box

[pixelspark]

Thought I'd post a small recipe because a lot of threads ask similar things.

Just sharing a setup that ended up working reasonably well: Phi, Ollama, Threadripper box, with a lot of attention on quant drift.

Feel free to tear it apart if you see something dumb.

[kernelmike]

Appreciate the detail. I ended up in about the same place using SGLang. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[plainline]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[torchdesk]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.

[latentgarden]

I'd start with the simpler stack.

[vectorpoint]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[visionyard23]

Need a little more detail here. Is this local-only or are you routing anything externally?

There are a few different conversations hiding inside this thread, so being explicit about the goal helps.

[rocmpath]

Can you fill in a bit more context? What model, what runtime, and roughly what context size?

[localcraft17]

subscribed, curious where this lands.

[thetanest]

following this because I'm in the same boat.