sqlite + embeddings surprisingly ok

[macrolab]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

I tested this tonight. On 7900 XTX with PyTorch, I saw basically the same behavior. The surprise was that memory pressure showed up before throughput collapsed.

[deepseeked]

Ran a quick check on my box. On mini PC with MLX, I saw basically the same behavior.

[phasetrack]

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Llama and compare from there. A lot of these threads end up being one bad config plus a pile of conclusions drawn from it.

[phasenode]

Tried to reproduce it just now. I couldn't reproduce the slowdown until I pushed context much higher.

[batchember]

I don't think that's the whole story. The reported numbers can look compelling, but in ordinary usage I didn't see the same benefit.

[bytegarden73]

Tried to reproduce it just now. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.