Formally, we define the task of compression as trying to retain as much performance as possible when compressing the officially released BERT-base (uncased) 5 to a 6-layer compact model with the same hidden size, following the settings in [28,<cite> 33,</cite> 37] .