simple router policy ideas

[northgarden37]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and M3 Max felt surprisingly usable once I tuned around reranking.

If someone wants the exact settings I can clean them up and post them.

For context, I'm not chasing leaderboard numbers. I mostly want the system to feel consistent day to day.

[stackpilot]

Good point. I ended up in about the same place using TensorRT-LLM. The biggest change for me was simplifying the stack instead of adding one more optimization layer.

[deepseeked]

Thanks, that helps. I ended up in about the same place using llama.cpp.

[audioriver]

Interesting, I hadn't tried that. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[macrolab]

Did a small comparison this afternoon. My result was mixed: first run looked bad, second run looked normal. The main thing that moved was latency, not quality.

[lablab]

following this because I'm in the same boat.