Pre-Trained Language Models:
We consider 4 types of transformer models—GPTNeo 125M, GPTNeo 1.3B, CodeGPT, and Codex  [1]}, [2]}, [3]}. We fine-tuned each of these pre-trained transformers on our Java dataset, except for Codex, for which we have no access to the pre-trained weights.
While our Nsg model only takes the headers in the Java class as inputs, for the various transformer models, the input is the entire Java class, including both headers and method bodies.
(We found that transformers perform quite poorly if only headers are provided.)
During evaluation, transformers are asked to predict the missing method body given the header and the rest of the class.
