where d is the number of columns of Q and K. In these work <cite>[34,</cite> 12, 7] , they all use the multi-head attention, as introduced in<cite> [34]</cite> , where matrices W Q i and In this work<cite> [34]</cite> , multiple groups of parameters (W Q i , W K i and W V i ) are used, which results in a large number of redundant parameters.