The reimplemented tree-to-tree and tree-to-seq models performed worse than the results reported by [1]}. This discrepancy could be caused by differences between the complexity of our datasets. Since their dataset was unavailable, we implemented a dataset with similar average program lengths. However, our dataset may have used more variables or constants, and it had much greater variation in program lengths. Our average program length was about the length of the programs in the easy version of their task, but our programs with longest length were about twice as long as the programs in the hard version of their task. It is also possible that despite our attempt to faithfully re-implement [1]} that there are slight differences between the two tree-to-tree models as we lacked their code. Any of those differences would be shared with the grammar model as except for the code corresponding to the decoder, they shared all of their code. To make it easier for future research to evaluate on the same task, we release our dataset and experiment code here, https://www.dropbox.com/sh/1q4aejr57jk40fs/AADKTvgKqLHuIIzNdlANjGRea?dl=0.
