There is an increasing interest in studying natural language and computer code together , as large corpora of programming texts become readily available on the Internet . For example , StackOverflow currently has over 15 million programming related questions written by 8.5 million users . Meanwhile , there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural language sentences . In this paper , we introduce a new named entity recognition ( NER ) corpus for the computer programming domain , consisting of 15,372 sentences annotated with 20 fine-grained entity types . We trained indomain BERT representations ( BERTOverflow ) on 152 million sentences from Stack-Overflow , which lead to an absolute increase of +10 F 1 score over off-the-shelf BERT . We also present the SoftNER model which achieves an overall 79.10 F 1 score for code and named entity recognition on StackOverflow data . Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERTbased tagging model . 1 Recently there has been significant interest in modeling human language together with computer code ( Quirk et al . , 2015 ; Iyer et al . , 2016 ; Yin and Neubig , 2018 ) , as more data becomes available on websites such as StackOverflow and GitHub . This is an ambitious yet promising direction for scaling up language understanding to richer domains . Access to domain-specific NLP tools could help a wide range of downstream applications . For example , extracting software knowledge bases from 1 Our code and data are available at : https:// github.com / jeniyat / StackOverflowNER/ text ( Movshovitz-Attias and Cohen , 2015 ) , developing better quality measurements of StackOverflow posts ( Ravi et al . , 2014 ) , finding similar questions ( Amirreza Shirani , 2019 ) and more . However , there is a lack of NLP resources and techniques for identifying software-related named entities ( e.g. , variable names or application names ) within natural language texts . In this paper , we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer programming domain . These named entities are often ambiguous and have implicit reliance on the accompanied code snippets . For example , the word ' list ' commonly refers to a data structure , but can also be used as a variable name ( Figure 1 ) . In order to recognize these entities , we propose a software-related named entity recognizer ( Soft-NER ) that utilizes an attention network to combine the local sentence-level context with corpuslevel information extracted from the code snippets . Using our newly annotated corpus of 15,372 sentences in StackOverflow , we rigorously test our proposed SoftNER model , which outperforms BiLSTM-CRF model and fine-tuned BERT model for identifying 20 types of software-related named entities . Our key contributions are the following : • A new StackOverflow NER corpus manually annotated with 20 types of named en-tities , including all in-line code within natural language sentences ( § 2 ) . We demonstrate that NER in the software domain is an ideal benchmark task for testing effectiveness of contextual word representations , such as ELMo ( Peters et al . , 2018 ) and BERT ( Devlin et al . , 2019 ) , due to its inherent polysemy and salient reliance on context . • An in-domain trained neural SoftNER tagger for StackOveflow ( § 3 ) that can recognize 20 fine-grained named entity types related to software developing . We also tested its performance on GitHub data of readme files and issue reports . • A code token recognizer ( § 3.1 ) that utilizes StackOveflow code snippets to capture the spelling patterns of code-related tokens , and consistently improves the NER tagger . • In-domain pretrained ELMo and BERT representations ( § 3.3 ) on 152 million sentences from StackOverflow that significantly outperforms off-the-shelf ELMo and leads to more than 21 points increase in F 1 score over offthe-shelf BERT . Overall , our named entity tagger ( SoftNER ) achieves a 79.10 % F 1 score on StackOverflow and 61.08 % F 1 score on GitHub data for extracting the 20 software related named entity types . We believe this performance is sufficiently strong to be practically useful . We have released our data and code , including the named entity tagger , our annotated corpus , annotation guideline , a specially designed tokenizer , and pre-trained StackOverflow BERT and ELMo embeddings . In this work , we investigated the task of named entity recognition in the social computer programming domain . We developed a new NER corpus of 15,372 sentences from StackOverflow and 6,510 sentences from GitHub annotated with 20 fine-grained named entities . We demonstrate that this new corpus is an ideal benchmark dataset for contextual word representations , as there are many challenging ambiguities that often require long-distance context to resolve . We also proposed a novel attention based model , named Soft-NER , that outperforms the state-of-the-art NER models on this dataset . Furthermore , we investigated the important sub-task of code recognition . Our code recognition model captures additional spelling information beyond then contextual word representations and consistently helps to improve the NER performance . We believe our corpus , StackOverflow-specific BERT embeddings and named entity tagger will be useful for various language-and-code tasks , such as code retrieval , software knowledge base extraction and automated question-answering . A Feature-Based CRF Baseline We implemented a CRF baseline model using CRFsuite 8 to extract the software entities . This model uses standard orthographic , contextual and gazetteer features . It also includes the code markdown tags ( § 3.1.3 ) and a set of regular expression features . The regular expressions are developed to recognize specific categories of coderelated entities . Feature ablation experiments on this CRF model are presented in Table 9
Resources and fundamental techniques are missing for identifying software-related named entities such as variable names or application names within natural language texts.
They introduce a manually annotated named entity corpus for the computer programming domain and an attention-based model which incorporates a context-independent code token classifier.
The proposed model outperforms BiLSTM-CRF and fine-tuned BERT models by achieving a 79.10 F1 score  for code and named entity recognition on their dataset which