To fine-tune on our Java dataset, we used the token-level code-completion task provided by CodeXGLUEhttps://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-token [1]}. During fine-tuning, the transformers are asked to predict the next token in an autoregressive fashion, just like in any language-modeling task. The learning rate is \(8e{-5}\)  and the batch size is 1 on each GPU. In total, we used 16 GPUs to fine-tune these transformers, which takes about 4 days to complete two epochs on our Java dataset, consisting of 600,000 training instances.
