distillation that keeps useful behavior: what are you seeing?

[quiettoken]

Thought I'd post a small recipe because a lot of threads ask similar things.

This isn't revolutionary, but the combination of Ollama and GB200 felt surprisingly usable once I tuned around tool calling.

Feel free to tear it apart if you see something dumb.

My use case is pretty ordinary: local chat, some document work, and occasional batch jobs.

[batchscope72]

What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

[thetagrid15]

What worked for me was pretty simple. If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious.

My first guess would be this: If you're on llama.cpp, I'd confirm the version first and then reduce variables until the behavior is obvious. Not glamorous advice, but repeatable baselines save a lot of time.

[turboyard94]

I'd start with the simpler stack.

[kernelghost]

Good point. I don't think your conclusion is far off. I got better results once I stopped chasing one metric and just looked at whether the workflow felt usable.

[quiettoken]

I don't think that's the whole story. I wouldn't optimize around that first. On my side the bigger swing came from the runtime and not the model. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet.