+- Inference (https://www.datasculpting.ai/inference)
+-- Forum: Harnesses, Agents & Orchestration (https://www.datasculpting.ai/inference/forumdisplay.php?fid=11)
+--- Forum: Agents (https://www.datasculpting.ai/inference/forumdisplay.php?fid=44)
+--- Thread: distillation that keeps useful behavior: setup advice (/showthread.php?tid=1217)
distillation that keeps useful behavior: setup advice - batchscope72 - 10-15-2026
Thought I'd post a small recipe because a lot of threads ask similar things.
This isn't revolutionary, but the combination of TensorRT-LLM and A6000 felt surprisingly usable once I tuned around speculative decoding.
If someone wants the exact settings I can clean them up and post them.
Re: distillation that keeps useful behavior: setup advice - northquill - 10-15-2026
A couple things I'd check first. Before rebuilding anything, I'd test one clean baseline with Gemma and compare from there. Not glamorous advice, but repeatable baselines save a lot of time.
Re: distillation that keeps useful behavior: setup advice - thetagrid15 - 10-17-2026
Not sure I agree there. I get why that seems right, but on A6000 the bottleneck felt elsewhere. Could be a workload difference, but I wouldn't generalize too fast from one setup.