Baselines.
We consider three categories of baselines.
The first consists of large pretrained transformers.
Specifically, we consider two variants of the GPT-Neo  [1]} model with 125 M and 1.3 B parameters. Both models are pre-trained on the Pile dataset [2]}, which consists of an 800 GB English-text corpus and open-source code repositories. On the APPS dataset [3]}, they perform well compared to OpenAI's 12-B-parameter, GPT-3-like Codex model [4]}. We also compare against CodeGPT  [5]} which is a GPT-2-like model with 125 million parameters. This model was pre-trained on Python and Java corpora from the CodeSearchNet dataset, which consists of 1.1 M Python functions and 1.6 M Java methods. We fine-tune all of these pretrained models on our Java dataset, using the token-level code-completion task provided by CodeXGLUE
 [5]}. Finally, we also offer a comparison against the Codex model [4]}.
Because we did not have access to the model's pretrained weights, this model is only used in a zero-shot fashion (no fine-tuning on our Java dataset). It should be noted here that the transformer baselines work on the entire Java language, whereas our Nsg framework works on a sub-part of Java which is supported in our grammar definition.
