distillation loss function

Model distillation, also known as knowledge distillation, is a supervised learning technique that condenses the capabilities and thought processes of a large, pre-trained “teacher” model into a smaller “student” model. This allows the student model to achieve comparable performance to the teacher model, but at a lower cost and with faster performance. Chinese AI lab … Read more