+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distillation that keeps useful behavior on 5090 (/showthread.php?tid=328)
distillation that keeps useful behavior on 5090 - vramridge - 09-18-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of llama.cpp and mini PC felt surprisingly usable once I tuned around router policy.
Feel free to tear it apart if you see something dumb.
Re: distillation that keeps useful behavior on 5090 - quietstack - 09-18-2026
My first guess would be this: Start by verifying the driver/runtime combo, then test with a smaller prompt, then look at token cost. Not glamorous advice, but repeatable baselines save a lot of time.
Re: distillation that keeps useful behavior on 5090 - llmtrack26 - 09-19-2026
I'd approach it in a boring order. On 5090, the easiest win is often to back off the aggressive settings and see whether the problem survives. Not glamorous advice, but repeatable baselines save a lot of time.