The vanilla sequence-to-sequence decoder predicts the tokens of the \(i{^{\mbox{\scriptsize th}}}\)  system response \(s_i\)  by first computing decoder hidden states via the recurrent unit. We denote \(\tilde{h}_1, \ldots , \tilde{h}_n\)  as the hidden states of the decoder and \(y_1, \ldots , y_n\)  as the output tokens. We extend this decoder with an attention-based model [1]}, [2]}, where, at every time step \(t\)  of the decoding, an attention score \(a_{i}^t\)  is computed for each hidden state \(h_i\)  of the encoder, using the attention mechanism of  [3]}. Formally this attention can be described by the following equations:
\(u_i^t &= w^T \tanh (W_2\tanh (W_1[h_i, \tilde{h}_t]))) \\a_i^t &= \textrm {Softmax}(u_i^t)\\\tilde{h}_t^{\prime } &= \displaystyle \sum _{i=1}^m a_i^th_i \\o_t &= U[\tilde{h}_t, \tilde{h}_t^{\prime }] \\y_t &= \textrm {Softmax}(o_t)\) 
