distillation that keeps useful behavior on 5090

[vramridge]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around router policy.

Feel free to tear it apart if you see something dumb.

[quietstack]

My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.

[llmtrack26]

I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.