Large pre-trained language models reach stateof-the-art results on many different NLP tasks when fine-tuned individually ; They also come with a significant memory and computational requirements , calling for methods to reduce model sizes ( green AI ) . We propose a twostage model-compression method to reduce a model 's inference time cost . We first decompose the matrices in the model into smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition . This approach has the benefit of reducing the number of parameters while preserving much of the information within the model . We experimented on BERTbase model with the GLUE benchmark dataset and show that we can reduce the number of parameters by a factor of 0.4x , and increase inference speed by a factor of 1.45x , while maintaining a minimal loss in metric performance . Deep learning models have been demonstrated to achieve state-of-the-art results , but require large parameter storage and computation . It 's estimated that training a Transformer model with a neural architecture search has a CO 2 emissions equivalent to nearly five times the lifetime emissions of the average U.S. car , including its manufacturing ( Strubell et al . , 2019 ) . Alongside the increase in deep learning models complexity , in the NLP domain , there has been a shift in the NLP modeling paradigm from training a randomly initialized model to fine-tuning a large and computational heavy pre-trained language model ( Howard and Ruder , 2018 ; Peters et al . , 2018 ; Devlin et al . , 2018 ; Radford , 2018 ; Radford et al . , 2019 ; Dai et al . , 2019 ; Yang et al . , 2019 ; Lample and Conneau , 2019 ; Liu et al . , 2019b ; Raffel et al . , 2019 ; Lan et al . , 2019 ; Lewis et al . , 2019 ) . While re-using pre-trained models offsets the training costs , inference time costs of the finetuned models remain significant , and are showstoppers in many applications . The main challenge with pre-trained models is how can we reduce their size while saving the information contained within them . Recent work , approached this by keeping some of the layers while removing others ( Sanh et al . , 2019 ; Sun et al . , 2019 ; Xu et al . , 2020 ) . A main drawback of such approach is in its coarse-grained nature : removing entire layers might discard important information contained within the model , and working at the granularity of layers makes the trade-off between compression and accuracy of a model hard to control . Motivated by this , in this work we suggest a more finegrained approach which decomposes each matrix to two smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition . This approach has the benefit of preserving much of the information while reducing the number of parameters . Alongside the advantage of preserving the information within each layer , there is also a memory flexibility advantage compared to removing entire layers ; As a result of decomposing each matrix to two smaller matrices , we can store each of the two matrices in two different memory blocks . This has the benefit of distributing the model matrices in many small memory blocks , which is useful when working in shared CPU-based environments . We evaluated our approach on the General Language Understanding Evaluation ( GLUE ) benchmark dataset ( Wang et al . , 2018 ) and show that our approach is superior or competitive in the different GLUE tasks to previous approaches which remove entire layers . Furthermore , we study the effects of different base models to decompose and show the superiority of decomposing a fine-tuned model compared to a pre-trained model or a ran-domly initialized model . Finally , we demonstrate the trade-off between compression and accuracy of a model . We presented a way to compress pre-trained large language models fine-tuned for specific tasks , while preserving much of the information contained within them , by using matrix decomposition to two small matrices . For future work it might be interesting to combine this approach with another approach such as pruning or quantization to achieve smaller models .
Existing coarse-grained approaches for reducing the inference time of pretraining models remove layers, posing a trade-off between compression and the accuracy of a model.
They propose a model-compression method which decompresses the matrix and performs feature distillation on the internal representations to recover from the decomposition.
The proposed method reduces the model size by 0.4x and increases inference speed by 1.45x while keeping the performance degradation minimum on the GLUE benchmark.