To evaluate our models and investigate the limitations of first order RNNs, we tested all models on recursive languages, including the Dyck languages, which require a stack-like structure as opposed to a simple counting mechanism in order to ensure successful recognition.
In addition, we also tested our models on a real-world problem, i.e., the Penn Treebank language modeling benchmark.
To our knowledge this is the first work to extensively compare a large variety of first order RNNs on complex CFLs task. We followed two training processes, one was sequential [1]}, [2]}, which is widely used in natural language processing related tasks, while the other was based on incremental learning [3]}, [4]} or curriculum learning [5]}, which is a popular training scheme for tensor RNNs. We compare these two variants in Table REF  and show that sequential prediction has a slight advantage over incremental learning.
<TABLE>