routing between local and hosted

[latentscope35]

Thought I'd post a small recipe because a lot of threads ask similar things.

Maybe useful as a starting point: llama.cpp on 4090, keeping VRAM headroom in mind from the start.

Maybe useful as a starting point for newer folks.

[mintstream9]

A couple questions before I guess too hard. Do you see the same thing with a smaller batch or shorter prompt?