distillation that keeps useful behavior with llama.cpp

[latentscope35]

Good point. I ended up in about the same place using vLLM. Worth rechecking your defaults though, because one bad assumption can make the rest of the thread look confusing. Turns out 'just one more test' was a lie.

[rocmmixer]

I think the bottleneck is somewhere else. I get why that seems right, but on Threadripper box the bottleneck felt elsewhere. Not saying you're wrong, just that I wouldn't lock that in as the lesson yet. The logs were somehow calmer than I was.