 A collection of scientific papers is often accompanied by tags:
  keywords, topics, concepts etc., associated with each paper.
   Sometimes these tags are human-generated, sometimes they are
  machine-generated.   We propose a simple measure of the consistency
  of the tagging of scientific papers: whether these tags are
  predictive for the citation graph links.   Since the authors tend to
  cite papers about the topics close to those of their publications, a
  consistent tagging system could predict citations.   We present an
  algorithm to calculate consistency, and experiments with human- and
  machine-generated tags.   We show that augmentation, i.e. the combination
  of the manual tags with the machine-generated ones, can enhance the
  consistency of the tags.   We further introduce cross-consistency,
  the ability to predict citation links between papers tagged by
  different taggers, e.g. manually and by a machine.
   Cross-consistency can be used to evaluate the tagging quality when
  the amount of labeled data is limited. A part of a construction of a knowledge graph is the analysis of publications and adding to them tags: concept names, keywords, etc. This often involves natural language processing or other machine learning methods BID8 . To develop such methods one must have a measure of success: one should be able to determine whether the given tagging is "good" or "bad". The most direct way to test the machine produced tags is to compare them to the tags produced by humans. One creates a "golden set" of papers tagged by humans, and penalizes the algorithms for any deviation from these tags. There are, however, certain problems with this approach. First, human tagging is expensiveeven more so for scientific papers, where human taggers must have a specialized training just to understand what the papers are about. Second, even the best human taggers' results are inconsistent. This provides a natural limitation for this method BID7 . The latter problem is exacerbated when the tagging dictionary is large. For example, the popular US National Library of Medicine database of Medical Subject Headings (MeSH, https://www.nlm.nih.gov/mesh/) has just under 30 000 entries. A superset of MeSH, Unified Medical Language System (UMLS, https://www.nlm.nih.gov/research/ umls/knowledge_sources/metathesaurus/release/statistics.html) contains a staggering amount of 3 822 832 distinct concepts. It is doubtful a human can do a good job choosing the right tags from a dictionary so large. A domain expert usually deals with a subsystem of the dictionary, covering her area of expertise. This presents obvious difficulties for tagging papers about multidisciplinary research, that may require a combination of the efforts of several highly qualified taggers. Another problem is the evaluation of tag augmentation. Suppose we have papers tagged by humans, and we want to add machine-generated tags, for example, to improve the search function in the collection. Do the new tags actually add to the quality or subtract from it? How can we evaluate the result if our tags are by definition different from those produced by humans?Thus a measure of the tagging quality other than a direct comparison with manually produced tags may be useful for the assessing the work of the tagging engines. This is especially important for an ongoing quality control of an engine that continuously ingests and tags fresh publications. In this paper we propose such a measure.The idea for this measure is inspired by the works on graph embeddings [Hamilton et al., 2018, Grover and BID3 . In these works one tags graph nodes and compares different sets of tags. The usual comparison criterion is whether the tags can predict graph edges: nodes connected by an edge should have similar tags, while nodes not connected by an edge should have dissimilar tags. To use this approach we need to represent papers as nodes on a graph. A natural choice is the citation graph: and edge from paper A to paper B means that paper A cites paper B. This leads to the following assumptions:1. Scientific papers cited by the given paper A are more similar to A than the other (non cited) papers.2. A good tagging system must reflect this.In other words, a good set of tags must be able to predict links on the citation graph, and the quality of the prediction reflects the quality of the tags. We will call this property consistency: a good tagger consistently gives similar tags to similar papers. It is worth stressing that consistency is just one component of the quality of a tagger. If a tagger consistently uses keyword library instead of keyword bread BID0 , this measure would give it high marks, despite tags being obviously wrong. A way to overcome this deficiency is to calculate cross-consistency with a known "good" tagger. For example, we can tag some papers manually, and some papers using machine generated tags, and then predict citation links between these papers. This cross-consistency measures the similarity between these taggers. This application is interesting because it allows us to expand the number of labeled papers for evaluation of machine-based taggers. We can create a golden set of manually tagged papers, and then generate tags for the papers in their reference lists, and the random samples using the machine-based tagger. Since a typical paper cites many publications , this approach significantly expands the quantity of data available for training and testing.To create a measure based on these ideas one should note that citation links strongly depend on the time the candidate for citation was published. Even a very relevant paper may not be cited if it is too old or too new. In the first case the citing authors may prefer a newer paper on the same topic. In the second case they may overlook the most recent publications. Therefore we recast our assumptions in the following way:A consistent tagging system should be able to predict citation links from a given paper to a set of simultaneously published papers.The rest of the paper is organized as follows. In Section 2 we discuss the algorithm to calculate the consistency of the given tagging system. Experiments with this measure are discussed in Section 3. In Section 4 we present the conclusions. First, there is clear difference between the consistency of the randomly generated tags and the real ones ( Figure 2 ). As expected, the consistency of the random tags is concentrated at AUC = 0.5, with some outliers both above and below this value. In contrast, the consistency of the real tags is almost always above AUC = 0.5. An exception is tagging sources of low coverage like GNAT (see Table 1 ), where consistency is close to 0.5. Obviously when the coverage is low, most positive and negative samples have zero overlap with their seed papers, which lowers AUC. Unexpectedly, the consistency of high coverage machine generated sources like NEJI is on par with the human tags.Tags augmentation is explored on Figure 3 . As expected, adding random tags to the manually generated ones does not noticeably change the consistency of the result. However, adding "real" machine generated tags is improving our measure, which is another evidence that the measure itself is reasonable.The cross-consistency between manual tags and machine-generated ones is shown on Figure 4 . Here we used different sources for seed papers and for samples. While crossconsistency is lower than the internal consistency of each tagger, is still is significantly higher than for random tags.In conclusion, a simple measure of consistency of tagging: whether it is predictive for citation links in a knowledge graph,-seems to be informative about the tagging process and can be used, along with other measures, to assess and evaluate it. Cross-consistency between different taggers can be used to estimate their similarity, especially when some taggers (e.g. manual tagging) are too expensive to run on a large set of papers. Cross consistency between manual tags and NEJI generated ones. X axis shows the source for the seed papers, Y axes shows the source for samples
A good tagger gives similar tags to a given paper and the papers it cites
{"src": "scitldr-AIC_train", "paper_id": "SyeD-b9T6m"}