Continuous vs Discrete Stacks: Using a continuous stack has been proven to very powerful when recognizing patterns generated by CFGs [1]}, [2]}, [3]}. However, it is still not entirely clear that continuous stack models are stable. Given that RNNs already in of themselves are difficult to train, adding a differentiable continuous stack introduces further instability during optimization, often resulting in sub-optimal performance. On the other hand, in theory, discrete stacks [4]}, [5]} are much more interpretable and stable.
Future work should be directed towards combining discrete optimization with continuous optimization in order to solve tasks that might benefit from the use of a stable, discrete stack.
