Casey Casalnuovo, Kenji Sagae, Prem Devanbu &ndash; University of California, Davis, USA
Studying the Difference Between Natural and Programming Language Corpora
University of California, Davis, USA
nlp/corpus-construction, nlp/text-corpora, programming-languages, nlp/syntax
The Germanand Spanish corpora were selected from a sample of files from the unlabeled datasets from the ConLL 2017 Shared Task (Ginter et al, 2017), which consist of web text obtained from CommonCrawl.⁸ Like the 1 billion token English corpus, we selected a random subsample to make these corpora size comparable with our other corpora. In this sample, we excluded files from the Wikipedia translations, as we observed Wikipedia formatting mixed in with some of the files.