Neural models of source code have received much attention in the recent past [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}.
However, these models have a basic weakness: while they frequently excel at
generating individual expressions or lines of code,
they do not do so well when tasked with synthesizing larger code blocks. For example, as we show later in this paper, state-of-the-art transformer models [9]}, [10]}, [8]}
can generate code with elementary semantic errors, such as uninitialized variables and type-incorrect expressions, when asked to generate method bodies, as opposed to single lines.
Even in terms of syntactic accuracy measures, the quality of the code that transformers produce on such “long-horizon” tasks can be far removed from the ground truth.
