@inproceedings{jin-etal-2024-align,
    title = "Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation",
    author = "Jin, Heegon  and
      Son, Seonil  and
      Park, Jemin  and
      Kim, Youngseok  and
      Noh, Hyungjong  and
      Lee, Yeonsoo",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.64",
    pages = "722--732",
    abstract = "The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation (NMT). Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the {``}Align-to-Distill{''} (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module (AAM) in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 DeâDsb and WMT-2014 EnâDe, respectively, compared to Transformer baselines.The code and data are available at https://github.com/ncsoft/Align-to-Distill.",
}

The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation (NMT). Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the {``}Align-to-Distill{''} (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module (AAM) in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 DeâDsb and WMT-2014 EnâDe, respectively, compared to Transformer baselines.The code and data are available at https://github.com/ncsoft/Align-to-Distill.
Jin, Heegon Son, Seonil Park, Jemin Kim, Youngseok Noh, Hyungjong Lee, Yeonsoo
Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation