The carbon footprint of natural language processing (NLP) research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation  to  improve  the  efficiency  of  the  Biaffine  dependency  parser  which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat & Manning, 2016).   When distilling to 20% of the original model’s trainable parameters, we only observe an average decrease of ∼1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.26x (1.21x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80% for some treebanks.   Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank. Ethical NLP research has recently gained attention (Kurita et al., 2019; Sun et al., 2019) . For example, the environmental cost of AI research has become a focus of the community, especially with regards to the development of deep neural networks (Schwartz et al., 2019; Strubell et al., 2019) . Beyond developing systems to be greener, increasing the efficiency of models makes them more cost-effective, which is a compelling argument even for people who might downplay the extent of anthropogenic climate change. In conjunction with this push for greener AI, NLP practitioners have turned to the problem of developing models that are not only accurate but also efficient, so as to make them more readily deployable across different machines with varying computational capabilities (Strzyz et al., 2019; Clark et al., 2019; Junczys-Dowmunt et al., 2018) . This is in contrast with the recently popular principle of make it bigger, make it better (Devlin et al., 2019; Radford et al., 2019) . Here we explore teacher-student distillation as a means of increasing the efficiency of neural network systems used to undertake a core task in NLP, dependency parsing. To do so, we take a state-of-theart (SoTA) Biaffine parser from Dozat & Manning (2016) . The Biaffine parser is not only one of the most accurate parsers, it is the fastest implementation by almost an order of magnitude among state-of-the-art performing parsers. Contribution We utilise teacher-student distillation to compress Biaffine parsers trained on a diverse subset of Universal Dependency (UD) treebanks. We find that distillation maintains accuracy performance close to that of the full model and obtains far better accuracy than simply implementing equivalent model size reductions by changing the parser's network size and training regularly. Furthermore, we can compress a parser to 20% of its trainable parameters with minimal loss in accuracy and with a speed 2.26x (1.21x) faster than that of the original model on CPU (GPU). We have shown the efficacy of using the teacher-student distillation technique for dependency parsing by distilling a state-of-the-art parser implementation. The parser used for our experiments was not only accurate but already fast, meaning it was a strong baseline from which to see improvements. We obtained parsing speeds up to 2.26x (1.21x) faster on CPU (GPU) while only losing ∼1 point for both UAS and LAS when compared to the original sized model. Furthermore, the smallest model which obtains these results only has 20% of the original model's trainable parameters, vastly reducing its environmental impact. A APPENDIX
We increase the efficiency of neural network dependency parsers with teacher-student distillation.
{"src": "scitldr-AIC_train", "paper_id": "Skx5ua4twS"}