Specifically, Ashish et.al<cite> [34]</cite> compute the attention function on a set of queries simultaneously, packed together into a matrix Q, while the keys and values are also packed together into matrices K and V , respectively. The attention function then adopts a no-linear function sof tmax over three matrices Q, K and V . There are two challenges to find a high-quality compression method to compress the multi-head attention in Transformer.