Carry Forward State Update: Whenever there exists a symbol that leads to more than one No-OP operation on the stack, we propose carrying forward the previous hidden state. Empirically, we found that this helps whenever the RNN encounters too many No-OPs in the input strings. This can be considered as approximation of a “reject” state, which have been shown to stabilize the learning process of higher order RNNs [1]}, [2]}. We keep a counter which indicates how many No-OPs the network has seen so far, which is used to trigger the update equation. With this counter-based trigger integrated, the full hidden state update equation can then be written as follows:
\(\mathbf {z}_t = u_t f_1(U \mathbf {x}_t + R \mathbf {\hat{z}}_{t-1}) + (1-u_t) \mathbf {\hat{z}}_{t-1}, \quad u_t = \bigg \lbrace \begin{array}{cc}u_t = 0, and ct>1 & \mbox{if NoOP} \\u_t = 1, & \mbox{otherwise} \\\end{array}\) 
