Multistage Feature Fusion Knowledge Distillation

This article discusses a new approach to knowledge distillation for improving the recognition accuracy of lightweight models. By using a cross-stage feature fusion symmetric framework, an attention mechanism, and a contrastive loss function, the proposed method effectively addresses the problem of significant differences in intermediate feature distributions between teacher and student models. Results show that this approach outperforms existing knowledge distillation methods on the CIFAR100 and TinyImagenet datasets.