where \(W_{xh}\in \mathbb {R}^{d_{model}\times h}, \)\(W_{hs}\in \mathbb {R}^{h\times s}\) , \(b_h\in \mathbb {R}^{1\times h}\)  and \(b_s\in \mathbb {R}^{1\times s}\)  are the parameters to be learned, \(h\)  is hidden size, \(s\)  is output size. \(ReLU\)  and \(tanh\)  is the activation function, \(x_t\)  is the tensor input at time \(t\) , and \(k_t\)  is the time gate [1]}.
