This architecture consists of a bidirectional RNN as an encoder (as seen in stage 1 of Figure ).
An input sentence is encoded in a sequence of annotations (one for each input word), corresponding to the concatenation of the outputs of a forward and a backward RNN.
Each annotation represents the full sentence with a strong focus on the current word.
The decoder is composed of a conditional RNN as provided for the DL4MT winter schoolhttps://github.com/nyu-dl/dl4mt-tutorial (see stage 3 of Figure ), equipped with an attention mechanism (stage 2).
The attention mechanism aims at providing weights for each annotation in order to generate a context vector (by performing a weighted sum over the annotations). The attention mechanism uses the hidden state at timestep \(j\)  of the decoder RNN along with the annotation \(h_i\)  to generate a coefficient \(e_{ij}\) . A softmax operation is performed over those coefficients to generate the annotation weights \(\alpha _{ij}\) .
As described in [1]}, the annotation weights can be used to align the input words to the output words.
The RNN takes as input the context vector, the embedding of the previous output word (stage 4), and of course its hidden state.
Finally, on stage 5 of the Figure , the output probabilities of the target vocabulary are computed.
The word with the highest probability is selected to be the translation at each timestep.
The encoder and the decoder are trained jointly to maximize the conditional probability of the correct translation.
