search and retrieval tuning tips

[sparrowhall]

Spent some time on this today and wanted to compare notes.

I've gone back and forth between MLX and llama.cpp for Mistral. The interesting difference for me has been quant drift.

What would you pick if you had to live with it for a few months?

[homelabhenry]

Appreciate the detail. The pattern around token cost has been pretty similar for me. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable. Turns out 'just one more test' was a lie.

[audionode8]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?

[audiopoint18]

subscribed, curious where this lands.

[orbitgarden]

Good point. The pattern around tool calling has been pretty similar for me.

[turbogrid11]

Good point. The pattern around tool calling has been pretty similar for me.

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?

[quantsupply]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

[routecrate]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

Good point. I don't think your conclusion is far off.

[batchshed]

Tried to reproduce it just now. The issue went away once I changed one setting and restarted the stack cleanly.

I tested this tonight. I couldn't reproduce the slowdown until I pushed context much higher.

[routeforge64]

I tested this tonight. The issue went away once I changed one setting and restarted the stack cleanly.