Results 21 to 30 of about 21,351 (259)
Faithful Knowledge Distillation
CoRR, 2023 Knowledge distillation (KD) has received much attention due to its success in compressing networks to allow for their deployment in resource-constrained systems. While the problem of adversarial robustness has been studied before in the KD setting, previous works overlook what we term the relative calibration of the student network with respect to its ...
Tom A. Lamb +5 more
openaire +2 more sources
Distilling Knowledge via Knowledge Review [PDF]
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021 CVPR ...
Pengguang Chen +3 more
openaire +2 more sources
Knowledge Diffusion for Distillation
Advances in Neural Information Processing Systems 36, 2023 The representation gap between teacher and student is an emerging topic in knowledge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific.
Tao Huang 0020 +6 more
openaire +3 more sources
Residual Knowledge Distillation
CoRR, 2020 9 pages, 3 figures, 3 ...
Mengya Gao +3 more
openaire +2 more sources
Relational Knowledge Distillation [PDF]
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher.
Wonpyo Park +3 more
openaire +2 more sources
CoRR, 2022 Recent studies pointed out that knowledge distillation (KD) suffers from two degradation problems, the teacher-student gap and the incompatibility with strong data augmentations, making it not applicable to training state-of-the-art models, which are trained with advanced augmentations.
Jihao Liu +3 more
openaire +2 more sources
What Knowledge Gets Distilled in Knowledge Distillation?
Advances in Neural Information Processing Systems 36, 2023 Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation.
Utkarsh Ojha +4 more
openaire +3 more sources
Knowledge Distillation for Quality Estimation [PDF]
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021 Quality Estimation (QE) is the task of automatically predicting Machine Translation quality in the absence of reference translations, making it applicable in real-time settings, such as translating online social media conversations. Recent success in QE stems from the use of multilingual pre-trained representations, where very large models lead to ...
Gajbhiye, A. +6 more
openaire +4 more sources
Multi-assistant Dynamic Setting Method for Knowledge Distillation [PDF]
Jisuanji kexueKnowledge distillation is increasingly gaining attention in key areas such as model compression for object recognition.Through in-depth research into the efficiency of knowledge distillation and an analysis of the characteristics of knowledge transfer ...
SI Yuehang, CHENG Qing, HUANG Jincai
doaj +1 more source
Interactive Knowledge Distillation
CoRR, 2020 Knowledge distillation is a standard teacher-student learning framework to train a light-weight student network under the guidance of a well-trained large teacher network. As an effective teaching strategy, interactive teaching has been widely employed at school to motivate students, in which teachers not only provide knowledge but also give ...
Shipeng Fu +5 more
openaire +2 more sources