In 2018, Bednarek et al. [1]} showed multiple instances of compilation errors in the original AlgoLisp dataset. Specifically, the DSL compiler fails to pass I/O pairs with ground truth code. They, therefore, constructed a filtered subset of AlgoLisp dataset containing only those problem instances that pass all the input-output test casesEven though we find few instances that resulted in partial passing of test cases.. Overall, the filtered dataset contains 90,153 instances. Table REF  also details the statistics of the filtered AlgoLisp dataset.
To the best of our knowledge, except NAPS [2]} and Karel [3]}, no similar code synthesis dataset exists that contains problem description along with the test cases and other meta information. Popular datasets like JAVA [4]}, WikiSQL [5]} only contain problem description and the corresponding code, leading to limitations in evaluating structurally different but logically similar synthesized codes. Although NAPS and Karel contains all the required meta-information, Karel does not deal with any natural language; it is a robotic programming language. On the other hand, in our internal data analysis, NAPS shows several data inconsistenciesThe NAPS dataset is very noisy due to crowd-sourcing. such as the presence of a long sequence of characters like abcdabcd, which conveys no meaning and inconsistent tokenization of sentences.
