@inproceedings{peck-brown-2024-tool,
    title = "Tool for Constructing a Large-Scale Corpus of Code Comments and Other Source Code Annotations",
    author = "Peck, Luna  and
      Brown, Susan",
    editor = "Gorman, Kyle  and
      Prud'hommeaux, Emily  and
      Roark, Brian  and
      Sproat, Richard",
    booktitle = "Proceedings of the Second Workshop on Computation and Written Language (CAWL) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.cawl-1.3",
    pages = "18--22",
    abstract = "The sublanguage of source code annotations{---}explanatory natural language writing that accompanies programming source code{---}is little-studied in linguistics. To facilitate research into this domain, we have developed a program prototype that can extract code comments and changelogs (i.e. commit messages) from public, open-source code repositories, with automatic tokenization and part-of-speech tagging on the extracted text. The program can also automatically detect and discard {``}commented-out{''} source code in data from Python repositories, to prevent it from polluting the corpus, demonstrating that such sanitization is likely feasible for other programming languages as well. With the current tool, we have produced a 6-million word corpus of English-language comments extracted from three different programming languages: Python, C, and C++.",
}

The sublanguage of source code annotations{---}explanatory natural language writing that accompanies programming source code{---}is little-studied in linguistics. To facilitate research into this domain, we have developed a program prototype that can extract code comments and changelogs (i.e. commit messages) from public, open-source code repositories, with automatic tokenization and part-of-speech tagging on the extracted text. The program can also automatically detect and discard {``}commented-out{''} source code in data from Python repositories, to prevent it from polluting the corpus, demonstrating that such sanitization is likely feasible for other programming languages as well. With the current tool, we have produced a 6-million word corpus of English-language comments extracted from three different programming languages: Python, C, and C++.
Peck, Luna Brown, Susan
Tool for Constructing a Large-Scale Corpus of Code Comments and Other Source Code Annotations