JSON mode across models

[cachebench]

This keeps coming up in conversation so I figured I'd start a thread.

One thing I keep noticing is that people underestimate reranking until they actually try to run this stuff for more than one afternoon.

Wondering if this is just where the whole space is heading now.

[papermixer4]

Tried to reproduce it just now. On 4090 with llama.cpp, I saw basically the same behavior.

[macroforge]

A couple questions before I guess too hard. Is this local-only or are you routing anything externally?