TRENDNERT: A Benchmark for Trend and Downtrend Detection in a Scientific Domain

Computational analysis and modeling of the evolution of trends is an important area of research in Natural Language Processing (NLP) because of its socio-economic impact. However, no large publicly available benchmark for trend detection currently exists, making a comparative evaluation of methods impossible. We remedy this situation by publishing the benchmark TRENDNERT, consisting of a set of gold trends and downtrends and document labels that is available as an unrestricted download, and a large underlying document collection that can also be obtained for free. We propose Mean Average Precision (MAP) as an evaluation measure for trend detection and apply this measure in an investigation of several baselines.


Introduction
Science changes and evolves rapidly: novel research areas emerge, while others fade away. Keeping pace with these changes is challenging. Therefore, recognizing and forecasting research trends is of great importance for researchers and academic publishers as well as for funding agencies, companies and journalists. The task of trend detection is often solved by domain experts who use special tools to investigate the data and get useful insights from it, e.g., through data visualization. However, manual analysis of large amounts of data can be time-consuming, and domain experts are expensive. Also, the overall increase of research publications (e.g., on Google Scholar, PubMed, DBLP) in the past decade makes an approach based on human domain experts increasingly difficult whereas automated approaches become a more desirable solution for the task of trend detection (Kontostathis et al. 2004).
At its inception, a new research topic often does not attract a lot of attention from the scientific community and is represented by just a few publications. He and Chen (2018) attribute the emergence of a trend to some triggering event, e.g., the publication of an article in a high-impact journal. Then the research topic starts to grow faster and becomes a trend (He and Chen 2018). We adopt this as our definition of trend in this paper: a trend is a research topic that has Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. a strongly increasing number of publications in a particular time interval. We define a downtrend as the converse: a downtrend is a research topic that has a strongly decreasing number of publications in a particular time interval. Downtrends can be as important to detect as trends, e.g., for a funding agency that is planning its budget.
There has been a lot of work on detecting trends in textual data, both in NLP and machine learning. However, despite the overall importance of trend analysis, no large publicly available benchmark for trend detection currently exists, making a comparative evaluation of methods impossible. Most of the prior work has used proprietary data or small datasets; or evaluation measures were not directly tied to the objectives of trend detection (Gollapalli and Li 2015).
We remedy this situation by publishing the benchmark TRENDNERT. 1 The benchmark consists of the following: 1. A set of gold trends compiled based on an extensive analysis of the metaliterature; 2. A set of labeled documents, where each document is assigned to a trend, a downtrend or to a third class, the flat topics; 3. An evaluation script to compute Mean Average Precision as the measure for the accuracy of trend detection algorithms; We make the benchmark available as an unrestricted download. 2 The underlying research corpus can be obtained for free from Semantic Scholar (Ammar et al. 2018). 3

Outline and Contributions
In this work, we address the challenge of benchmark creation for (down)trend detection. This paper is structured as follows: Section 2 discusses related work. Section 3 introduces our model for corpus creation and its components. In Section 4, we present the crowdsourcing procedure for labeling the benchmark. Section 5 introduces our evaluation measure. Section 6 describes the proposed baseline for trend detection, our experimental setup, and final results. Finally, we present the details of the distributed benchmark in Section 7 and conclude in Section 8. Our contributions are as follows: • TRENDNERT is the first publicly available benchmark for (down)trend detection.
• TRENDNERT is based on a collection of more than a million documents. It is among the largest that has been used for trend detection and therefore offers a realistic setting for developing trend detection algorithms.
• TRENDNERT addresses the task of detecting both trends and downtrends. To the best of our knowledge, the task of downtrend detection has not been addressed before.

Related Work
In addition to growth, two other characteristics of trends that have been proposed are novelty and utility, and some prior work has used this more narrow definition of trend (Tu and Seng 2012;Small, Boyack, and Klavans 2014;Rotolo, Hicks, and Martin 2015;He and Chen 2018). Unfortunately, novelty and utility are much harder to quantify automatically than growth. For this reason, we define trends in terms of growth only in this paper, i.e., a trend is a research topic that has a strongly increasing number of publications in a particular time interval. Prior work on analyzing topics and their evolution over time can be classified according to the primary source of information it employs: text, citations and key phrases.
Text-based analysis: There has been a great deal of prior work on text-based analysis for trend detection and evolution (Blei and Lafferty 2006;Wang and McCallum 2006;Hall, Jurafsky, and Manning 2008;Gollapalli and Li 2015;Gupta et al. 2018;Schein et al. 2019). However, our focus in this paper is not to develop new algorithms or models, but instead to create a benchmark. We have therefore selected a simple baseline text-based analysis method for trend detection.
Citation-based analysis is the second popular direction that is considered to be effective for trend identification (Le, Ho, and Nakamori 2005;Small 2006;He et al. 2009;Shibata et al. 2008;Shibata, Kajikawa, and Takeda 2009). This method assumes that citations in their various forms (bibliographic citations, co-citation networks, citation graphs) indicate meaningful relationships between topics and uses citations to model the topic evolution in the scientific domain. Nie and Sun (2017) utilize this approach along with Latent Dirichlet Allocation (LDA) and k-means clustering to identify research trends. The authors first use LDA to extract features and determine the optimal number of topics. Then they use k-means to obtain thematic clusters, and finally, they compute citation functions to identify the changes of clusters over time. Though the main drawback of citationbased analysis is that there are many trend detection scenarios (e.g., research news articles or research blog posts) in which citations are not available and the approach is therefore not applicable.
Keyphrase-based analysis: This approach is based on keyphrase information extracted from research papers. A keyphrase is interpreted as a representative for a single research topic. For example, Asooja et al. (2016) utilize the Saffron system (Monaghan et al. 2010) to extract keywords from LREC proceedings and then forecast trends based on regression models. A potential problem with this approach is that there is no clear correspondence between topics and keyphrases. Keyphrases are noisy, ambiguous (e.g., Java as an island vs. Java as a programming language), and many topics may not correspond to a single keyphrase because there are several equivalent names for a topic.

Corpus Creation
In this section, we describe the methodology we use to create the TRENDNERT benchmark.

Underlying Data
We use a subset of the Open Research Corpus provided by Semantic Scholar. It contains more than one million papers published mostly between 2000 and 2015 in about 5000 computer science journals and conference proceedings. 4 Each document in the collection consists of title, keyphrases, abstract and other metadata (e.g., venue, year).

Stratification
The distribution of papers over time in the underlying dataset is skewed: years before 2000 have less than 1000 documents per year (see Figure 1). 5 During our experiments, we found that clustering the entire collection or a random sample does not support high-quality identification of (down)trend candidates, because more weight is given to later years than to earlier years (see Section 6.2). Therefore, the first step in the benchmark creation was the generation of a stratified sample of the original document collection. To this end, we randomly select 10, 000 documents for each year between 2000 and 2015 for an overall sample size of 160, 000.

Document Representations & Clustering
As we mentioned before, our main focus in this paper is not to develop new algorithms or models, but instead to create a benchmark. We have therefore selected an algorithm based on k-means clustering as a simple baseline method for trend detection. In traditional document clustering (Manning, Raghavan, and Schütze 2008), documents are usually represented as bag-of-words (BOW) feature vectors that however have a major weakness: they ignore the semantics of words (Le and Mikolov 2014). Still, the recent work in representation learning and in particular doc2vec -the method proposed by Le and Mikolov (2014) -can provide representations to document clustering that overcome this weakness. The doc2vec 6 algorithm is inspired by techniques for learning word vectors (Mikolov et al. 2013) and is able to capture semantic regularities in document collections. It is an unsupervised approach for learning continuous distributed vector representations for pieces of text (i.e., paragraphs, documents). This approach maps texts into a vector space such that semantically similar documents are assigned similar vector representations (e.g., an article about genomics is closer to an article about gene expression than to an article about fuzzy sets). Other work has already successfully applied this type of document representations for topic modeling, combining them with both LDA and clustering approaches ( In our work we run doc2vec on the stratified sample of 160, 000 papers and represent each document as a lengthnormalized vector (i.e., as a document embedding). These vectors are then clustered into k = 1000 clusters using the scikit-learn 7 implementation of k-means (MacQueen 1967) with default parameters. The combination of document representations with a clustering algorithm is conceptually simple and interpretable. Also, the comprehensive comparative evaluation of topic modeling methods utilizing document embeddings performed by Curiskis et al. (2019) showed that doc2vec feature representations with k-means clustering outperform several other methods 8 on three evaluation measures (Normalized Mutual Information, Adjusted Mutual Information, and Adjusted Rand Index).
We run 10 trials of stratification and clustering, resulting in 10 different clusterings. We do this to protect against the variability of clustering and because we do not want to rely on a single clustering for proposing (down)trends for the benchmark. 6 We use the implementation provided by Gensim: https:// radimrehurek.com/gensim/models/doc2vec.html 7 https://scikit-learn.org/stable/ 8 hierarchical clustering, k-medoids, NMF and LDA

Trend & Downtrend Estimation
Recall our definitions of trend and downtrend. A trend (resp. downtrend) is a research topic that has a strongly increasing (resp. decreasing) number of publications in a particular time interval. Linear regression is a widely used technique in trend analysis (Hess, Iyer, and Malm 2001) that can identify changes over time in a variable. It correlates the measurements of the variable to the times at which they occurred. Here, we use linear regression to identify trend and downtrend candidates in a given set of clusters. We count the number of documents per year for a cluster and then estimate the slope of the best fitting line. Clusters are then ranked according to the resulting slope and the n = 150 clusters with the largest (resp. smallest) slope are selected as trend (resp. downtrend) candidates. Thus, our definition of a singleclustering (down)trend candidate is a cluster with extreme ascending or descending slope. Figure 2 gives two examples of (down)trend candidates.

(Down)trend Candidate Validation
We run ten rounds of clustering (see. Section 3.3), resulting in ten different clusterings. We then estimate the (down)trend candidates according to the procedure described in Section 3.4, so that for each of the ten clusterings, we obtain 150 trend and 150 downtrend candidates.
For our benchmark, among all the (down)trend candidates across the ten clusterings, we want to keep only those that are consistent across clusterings. To obtain such consistent (down)trend candidates, we employ the Jaccard coefficient. We define an equivalence relation ∼ R : two candidates (i.e., clusters) are equivalent if their Jaccard coefficient is ≥ τ ∼ , where τ ∼ = 0.5. We compute the equivalence classes of ∼ R over the ten clusterings. Finally, we define an equivalence class as an equivalence class trend candidate (resp. equivalence class downtrend candidate) if it contains trend (resp. downtrends) candidates in at least half of the clusterings. This procedure gave us in total 110 equivalence class trend candidates and 107 equivalence class downtrend candidates that we then annotate for our benchmark. 9

Benchmark Annotation
To annotate the equivalence class (down)trend candidates obtained from our model we used the Figure-Eight 10 platform. It provides an integrated quality-control mechanism and a training phase before the actual annotation (Kolhatkar, Zinsmeister, and Hirst 2013), which helps disqualify poorly performing annotators.
We design our annotation task as follows. A worker is shown a descriptor of a particular candidate along with the list of possible (down)trend tags (i.e., "curated" names). The descriptors of candidates are obtained through the following procedure: • First, we examine the papers assigned to a candidate cluster.
• Then we extract keywords and titles of the papers assigned to the cluster. Semantic Scholar provides keywords and titles as a part of the metadata in the underlying collection (see Section 3.1).
• Finally, we compute the top 15 most frequent keywords and randomly pick 15 titles. These keywords and titles are the descriptor of the candidate shown to crowd-workers.
We create the (down)trend tags as follows. Based on keywords, titles and metadata such as publication venue, cluster candidates are manually labeled by graduate employees (i.e., domain experts in the computer science domain). Examples of (down)trend tags created this way are "XML" and "Sentiment and Emotion Analysis (using Social Media Channels)".
In the crowdsourcing interface, the descriptor of a (down)trend candidate is shown on the right. The (down)trend tags are shown on the left. They are mixed and presented in random order. The temporal presentation order of candidates (i.e., descriptors) is also random. The worker is asked to pick from the list of (down)trend tags the tag that 9 Note that the overall number of 217 refers to the number of the (down)trend candidates and not to the number of documents. Each candidate contains hundreds of documents. 10 formerly called CrowdFlower matches the descriptor best. Even if there are several tags that are a possible match, the worker must choose one. If there is no matching tag, the worker can select the option other. Three different workers evaluate each (down)trend candidate (i.e., descriptor), and the final label is the majority label. If there is no majority label, we present the (down)trend candidate repeatedly to the workers until there is a majority.

Inter-annotator Agreement
To measure the quality of annotation, we compute the interannotator agreement -a measure of how well two (or more) annotators agree. We use two measures of inter-annotator agreement: Krippendorff's α and the Figure-Eight Internal Agreement Rate.
Krippendorff's α is a reliability coefficient developed to measure the agreement between observers (i.e., annotators) (Krippendorff 2011). We choose Krippendorff's α as an inter-annotator agreement measure because it -unlike other specialized coefficients -is a generalization of several reliability criteria and applies to situations like ours where we have more than two annotators and a given annotator only annotates a subset of the data (i.e., we have to deal with missing values). For our gold standard, Krippendorff's α for the 217 annotated down(trend) candidates (represented as descriptors) is α = 0.798. According to Landis and Koch (1977), this value corresponds to a substantial level of agreement.  Figure-Eight provides an agreement score c for each annotated unit u, which is based on the majority vote of the trusted workers. The score has been proven to perform well compared to classic metrics (Kolhatkar, Zinsmeister, and Hirst 2013). For our gold standard, the average c for the 217 annotated (down)trend candidates is 0.799.
Since both Krippendorff's α and the Figure-Eight Internal Agreement Rate are quite high, we consider the obtained annotations for the benchmark to be of good quality.

Gold Trends
We then compiled a list of computer science trends for the year 2016 based on our analysis of twelve survey publications (Augustin 2016;Frot 2016;Brooks 2016;Reese 2015;Markov 2015;Zaino 2016;Rivera 2016;Harriet 2015;IEEE Computer Society 2016;Nessma 2015;Meyerson and Mariette 2016;Ankerholz 2016). We identified a total of 31 trends; see Table 1. Since there is variation as to the name of a specific trend, we created a standard name for each. Of the 31 trends, 28 (90%) were found as trend candidates during crowdsourcing 11 . We searched for the three missing trends using a variety of techniques: inspection of non-trends/downtrends; random browsing of documents not assigned to trend candidates; and keyword searches. Two of the missing trends do not seem to occur in the collection at all: Human Augmentation and Food and Water Technology. The third missing trend is Cryptocurrency and Cryptography; it does occur in the collection, but the occurrence rate is very small (ca. 0.0001). We do not consider these three trends in the rest of the paper.
In summary, based on our analysis of the meta-literature, we identified 28 gold trends that also occur in our collection. We will use them as the gold standard that trend detection algorithms should aim to discover.

Evaluation Measure for Trend Detection
Whether something is a trend is a graded notion. For this reason, we adopt an evaluation measure that is based on a ranking of trend candidates: Mean Average Precision (MAP). The score represents the average of the precision values of ranks of correctly identified (and non-redundant) trends.
We define trend detection as computing a ranked list of sets of documents (i.e., papers), where each set is a trend candidate. We refer to documents as trendy, downtrendy or flat depending on which category they are assigned to in our benchmark. A trend candidate is called trendy, downtrendy or flat, depending on which of the three classes constitutes the plurality of documents. We count a trend candidate c as a correct recognition of gold trend t if it meets the following conditions: • c is trendy; • t is the largest trend in c; • |t ∩ c|/|c| ≥ ρ, where ρ is a coverage parameter; • t was not earlier recognized in the list.
This criterion give us a true positive (recognition of a gold trend) or false positive (otherwise) for each position of the ranked list, and a precision score for each trend that was found. Precision for a trend that was not found is 0. We then compute MAP; see Table 2 for an example.

Trend Detection Baselines
In this section, we investigate the impact of four different configuration choices on the performance of trend detection by way of ablation.

Configuration Choices
Abstracts vs. Full text. Our subset of data collection 12 contains both abstracts and full text documents. Our initial expectation was that the full text of a document contains more information than the abstract and should be a better basis for trend detection. This is one of the configuration choices we test in the ablation. We apply our proposed method on full text (document content without abstract) and abstract (only abstract) collections separately and observe the results.
Measure of Growth: Length L t of Interval. Clusters are ranked by trendiness, and the resulting ranked list is evaluated. To measure the trendiness or growth of topics over time we fit a line to an interval {(i, n i }|i 0 ≤ i < i 0 + L t } by linear regression, where L t is the length of the interval, i is one of the 16 years (2000, . . . , 2015) and n i is the number of documents that were assigned to cluster c in that year.
As a simple default baseline, we apply the regression to half the entire interval, i.e., L t = 8. There are nine such intervals in our 16-year period. As the final measure of growth for the cluster, we take the max of the nine individual slopes.
To determine how much this configuration choice affects our results, we also test a linear regression over the entire time span, i.e., L t = 16. In this case, there is a single interval.
Clustering Method. We consider Gaussian Mixture Models (GMM) and k-means. We use GMM with spherical covariance, where each component has the same variance. For both, we proceed as described in Section 3.3.

Experimental Setup and Results
We conducted experiments on the Semantic Scholar corpus and evaluated a ranked list of trend candidates against the benchmark. We used MAP as the primary evaluation measure. As a secondary evaluation measure, we compute recall at 50 (R@50), which estimates the percentage of gold trends found in the list of 50 highest-ranked trend candidates. We found that the configuration (0) in Table 3 works best. We then conducted ablation experiments to determine the importance of the configuration choices.
Comparing (0) and (1), 13 we see that abstracts (A) are a better representation for our trend detection baseline than full text (F ). This can be explained by the fact that an abstract is a summary of a scientific paper that covers only the central points, and thus it is semantically very concise and rich. In contrast, the full text contains many parts that are secondary (e.g., future work, related work) and thus may be an inferior representation of the overall meaning of the document.
Comparing (0) and (2), we observe that stratification (y) clearly improves results compared to the setting with unstratified data (n), i.e., randomly sampled data. We would   expect the effect of stratification to be even stronger for collections in which the distribution of documents over time is more skewed than in ours.
Comparing (0) and (3), the length of the interval is an important configuration choice: as we expected, the 16 year interval is too long and the 8 year interval seems to be a better choice, especially if one aims to find short-term trends.
Comparing (0) and (4), we see that topics obtained from k-means and from GMM have similar performance. How-ever, GMM, which takes variance into account and estimates a soft assignment, performs slightly better.

Benchmark Description
We now describe the contents of the TRENDNERT benchmark. It contains the following components.
1. Two files containing information about documents, one for documents assigned to (down)trends and one for documents assigned to flat topics, i.e., topics that are neither trends nor downtrends; 2. A script for generating document hash codes; 3. A file mapping hash codes to our internal IDs; 4. A script for computing MAP for a list of (down)trend candidates (default setting is ρ = .25); The two files containing information about documents give the following information for each document. • Paper ID: the internal ID we use for documents in the Semantic Scholar collection; • Cluster ID: unique ID of the equivalence class in which the document occurs; • Label: (down)trend tag assigned by crowd-workers (e.g., Cyber Security, Fuzzy Sets and Systems); • Type of (down)trend candidate: trend (T ), downtrend (D) or flat topic (F ); • Hash ID 14 of each paper from the Semantic Scholar collection;

Conclusion
With this work we release TRENDNERT -the first publicly available benchmark for (down)trend detection that offers a realistic setting for evaluating trend detection algorithms.
(  TRENDNERT also supports downtrend detection -an important problem that was not addressed before. We evaluate a number of baselines for trend detection on TREND-NERT. We find that, for these baselines, stratification improves trend detection if the distribution of documents is skewed and that abstract-based representations perform better than fulltext representations for the models included in our experiments.