08-28-2026, 05:00 AM
Thought I'd post a small recipe because a lot of threads ask similar things.
Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, 4090, with a lot of attention on reranking.
Feel free to tear it apart if you see something dumb.
This is mostly for local work rather than public serving.
Just sharing a setup that ended up working reasonably well: Mistral, llama.cpp, 4090, with a lot of attention on reranking.
Feel free to tear it apart if you see something dumb.
This is mostly for local work rather than public serving.
measure twice, benchmark three times