Search / Retrieval / Reranking discussion with SGLang

[llmtrack]

Spent some time on this today and wanted to compare notes.

I'm comparing two approaches here: keep it simple on mini PC, or squeeze more out of the stack and deal with the complexity.

What would you pick if you had to live with it for a few months?

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[fusedhall]

Spent some time on this today and wanted to compare notes.

A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Qwen and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.

[turboscope]

I'd approach it in a boring order. Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at embeddings. Not glamorous advice, but repeatable baselines save a lot of time.