+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distillation that keeps useful behavior with llama.cpp (/showthread.php?tid=788)
Pages:
1
2
Re: distillation that keeps useful behavior with llama.cpp - latentscope35 - 10-10-2026
Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. Turns out 'just one more test' was a lie.
Re: distillation that keeps useful behavior with llama.cpp - rocmmixer - 10-11-2026
I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. The logs were somehow calmer than I was.