+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distillation that keeps useful behavior: what are you seeing? (/showthread.php?tid=915)
distillation that keeps useful behavior: what are you seeing? - quiettoken - 10-09-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of Ollama and GB200 felt surprisingly usable once I tuned around tool calling.
Feel free to tear it apart if you see something dumb.
My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.
Re: distillation that keeps useful behavior: what are you seeing? - batchscope72 - 10-09-2026
What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
Re: distillation that keeps useful behavior: what are you seeing? - thetagrid15 - 10-09-2026
Quote:What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.
My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.
Re: distillation that keeps useful behavior: what are you seeing? - turboyard94 - 10-09-2026
I'd start with the simpler stack.
Re: distillation that keeps useful behavior: what are you seeing? - kernelghost - 10-10-2026
Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.
Re: distillation that keeps useful behavior: what are you seeing? - quiettoken - 10-11-2026
I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.