We evaluated our approach by accuracy and BLEU scores. Ideally, the accuracy should count the fraction of functionally correct programs, which unfortunately is not Turing computable. We followed most previous studies [1]}, [2]}, and calculated the accuracy based on string match (denoted as StrAcc).Since spaces and empty lines are not represented in AST, the string match is computed based on a “normalized” format. Notice that indentation is crucial to a python program, which has been implicitly captured by AST. We also find that several generated programs use a different variable name but implements a correct functionality, and that sometimes an argument name in a function call is or is not specified. Although different from the reference program, they are obviously correct programs after manual inspection, and we denote human-adjusted accuracy by Acc+. Here, we did not perform checking for non-obvious alternative implementation of an algorithm, and thus Acc+ is still a lower bound of functional accuracy.
