Deep Models of Code.
There is, by now, a substantial literature on deep models trained on program syntax. Early work on this topic
represented programs as sequences [1]} or trees [2]}, [3]}, [4]}, and learned using classic neural models, such as RNNs,
as well as specialized
architectures [5]}, [6]}, [7]}.
The recent trend is to use transformers [8]}, [9]}, [10]}, [11]}.
Some of these models — for example, CodeGPT  [11]} — are trained purely on code corpora (spanning a variety of languages, including Java). Other models, such as CodeBert [10]}, GPT-Neo  [14]}, and Codex  [15]}, are trained on both natural language and code. In all of these cases, programs are generated
without any explicit knowledge of program syntax or semantics.
