The input data \(X\)  may contain several levels of correlation information and the learning process may benefit from processing the input data in multiple different ways. Multiple self-attention heads are introduced that operate on the same input in parallel and use distinct weight matrices \(W_q, W_k,\)  and \(W_v\) , to extract various levels of correlation between the input data. For example, consider the sentence “Do we have to turn left from here or have we left the street behind?”. There are two occurrences of the word “left” in the sentence. Each occurrence has a different meaning, and consequently a different relationship with the rest of the words in the sentence. Transformers can capture such information by using multiple heads as shown in Figure REF . Each head is built using a separate set of query, key and value weight matrices and computes self-attention over the input sequence in parallel with other heads. The use of multiple heads in Transformer is analogous to using multiple kernels at each layer in convolutional neural network, where each kernel is responsible for learning distinct features or representations [1]}.
<FIGURE><FIGURE>