We evaluate our approach in the task of generating the entire body of a Java method given the rest of the class in which the method occurs.
We consider a large corpus of curated Java
programs, over a large vocabulary of API methods and types.Using this corpus, we train an Nsg whose attributes, among other things, track the state of the symbol table and the types of arguments and return values of invoked methods at various points of a program,
and whose neural component is a basic tree LSTM.
We compare this model against several recent models: fine-tuned versions of two GPT-Neo  [1]} transformers and the CodeGPT  [2]} transformer, OpenAI's Codex system [3]} (used in a zero-shot manner),
and a GNN-based method for program encoding
[4]}. Some of these models are multiple orders of magnitude larger than our Nsg model.
Our experiments show that the Nsg model reliably outperforms all of the baselines on our task,
both in terms of producing programs free of semantic errors and in terms of matching the ground truth syntactically.
