Following prior work, we pretrain CoditT5 on large amounts of source code and natural language comments from CodeSearchNet [1]} dataset which consists of functions of six programming languages (java, python, ruby, php, go and javascript) together with the human-written comments.
CodeSearchNet dataset is widely used to pretrain large language models, such as CodeT5  [2]} and UniXcoder [3]}.
We use the processed CodeSearchNet dataset
provided by [3]} which contains 6.1 million programming languages data and 1.9 million natural language data.
