Computer Science
Computer Science-2021.gz:143200
Computer Science
1ced0a5af5fab24b25e624a1c09c7e1f6fd657ca
Text Summarization of Czech News Articles Using Named Entities

The foundation for the research of summarization in the Czech language was laid by the work of Straka et al. (2018). They published the SumeCzech, a large Czech news-based summarization dataset, and proposed several baseline approaches. However, it is clear from the achieved results that there is a large space for improvement. In our work, we focus on the impact of named entities on the summarization of Czech news articles. First, we annotate SumeCzech with named entities. We propose a new metric ROUGE_NE that measures the overlap of named entities between the true and generated summaries, and we show that it is still challenging for summarization systems to reach a high score in it. We propose an extractive summarization approach Named Entity Density that selects a sentence with the highest ratio between a number of entities and the length of the sentence as the summary of the article. The experiments show that the proposed approach reached results close to the solid baseline in the domain of news articles selecting the first sentence. Moreover, we demonstrate that the selected sentence reflects the style of reports concisely identifying to whom, when, where, and what happened. We propose that such a summary is beneficial in combination with the first sentence of an article in voice applications presenting news articles. We propose two abstractive summarization approaches based on Seq2Seq architecture. The first approach uses the tokens of the article. The second approach has access to the named entity annotations. The experiments show that both approaches exceed state-of-the-art results previously reported by Straka et al. (2018), with the latter achieving slightly better results on SumeCzech's out-of-domain testing set.


Introduction
Automatic text summarization is an important task of natural language understanding. The goal is to describe a text accurately, be it a news article, a web page, or a paragraph of a book, using shorter text. The shorter text can be in the form of a paragraph, sentence, or even a few words. Automatic text summarization is a challenging problem for automatic systems because they have to excel in multiple areas at once. They have to understand the meaning of the original text, understand which passages are important and which can be excluded, and generate meaningful and grammatically correct summarizations.
In this work, we focus on the summarization of Czech news articles by a one-sentence summary. We can also describe this task as the automatic creation of a headline for a given text. We use the SumeCzech dataset (Straka et al., 2018) for our experiments.
Additionally, we explore the influence of the named entities on text summarization. We use SpaCy's named entity recognition (NER) model, trained on a CoNLL-based extended CNEC 2.0 dataset (Ševčíková et al., 2014), to label SumeCzech with named entities to create additional features. We publish the annotations to promote the replication of results and to enable further research.
We use the annotations as a foundation for our newly proposed Named Entity Density. The method selects the sentence with the highest ratio of the number of entities to the sentence length as the summary of the article. We show that our proposed method achieves nearly as good results in the automatic evaluation as the hard-to-beat baseline in the news domain that selects the first sentence. Nenkova (2005) shows that the baseline selecting the first sentence is a strong baseline because authors tend to summarize the main points of an article in the first sentence, especially in the news domain. We also show that sentences selected by Named Entity Density possess a high information value mentioning to whom, where, when, and what happened. This structure resembles the style of reports that concisely identifies and examines issues, events, or findings that have happened. We propose that such a summary is useful in voice applications presenting news. Voice application can present the summary formed out of the first sentence of a news article first and continue with the sentence selected by Named Entity Density if a user requests additional information.
We also propose two abstractive methods that can construct a novel sentence as a summary. They are based on the Seq2Seq architecture, initially used for machine translation. The first method uses the text of the article only. The second method uses additional annotations created by the name entity recognition system as input features. Our experiments show that both models achieved state-of-the-art results in SumeCzech's task to summarize the headline from the text of the article. We also show that the named entities added as an additional input feature improve the ability of the model to generalize to the out-of-domain data.
Finally, we propose a new metric Rogue NE , which measures the overlap of named entities in the target and generated summaries. Poor results of the experiments in Rogue NE show that summarization of entities still poses many challenges, and this task has not been solved yet. Allahyari et al. (2017) provide a brief survey of text summarization. In general, we divide text summarization algorithms into two categories, extractive and abstractive.

Extractive Summarization
Extractive summarization algorithms choose pieces from the original text, usually sentences, and combine them to form a summary. From a high-level perspective, most extractive summarizers follow the same two steps: First, score all sentences. Then, pick N sentences with the highest score. The main difference between individual extractive methods is how they score sentences. The advantage of extractive methods is that no matter how simple the method is, it always produces syntactically correct sentences, even though they may not be useful summaries. On the other hand, there is a disadvantage too. Extractive summarizers are limited in what they can predict by the sentences of the source text. Thus, more elaborate summaries are out of their reach. Mihalcea and Tarau (2004) introduce a Textrank algorithm, a graph-based ranking model. It creates a graph of sentences based on their overlap. It chooses the most important sentences according to the created graph. Pal and Saha (2014) propose a summarization algorithm that derives the relevance of the sentences within the text using the Simplified Lesk algorithm and the WordNet online database. Kågebäck et al. (2014) propose using continuous vector representations for semantically aware representations of sentences for summarization. Zhang et al. (2016) develop convolutional neural networks that learn sentence features and perform sentence ranking. The latest results are achieved by Liu (2019). They apply the BERT model (Devlin et al., 2018) to extractive summarization.
Especially relevant works for our research are those working with named entities. Nobata et al. (2002) introduce named entity tagging and pattern discovery to a summarization system based on a sentence extraction technique. Hassel (2003) integrates a Named Entity tagger into the SweSum summarizer for Swedish newspaper texts. Filatova and Hatzivassiloglou (2004) propose a summarization technique using a set of features based on low-level, atomic events that describe the relationships between important actors in a document or in a set of documents. The extraction of atomic events relies on a noun phrase and named entity recognition (Hatzivassiloglou and Filatova, 2003). Jabeen et al. (2013) apply named entity recognition for summarization of tweets. Schulze and Neves (2016) present EntityRank, a multidocument graphbased summarization algorithm that is solely based on named entities. They apply it to texts from the medical domain successfully. Khademi and Fakhredanesh (2020) propose an unsupervised method for summarizing Persian texts that use a named entity recognition system. Their method consists of three phases: training a supervised NER model, recognizing named entities in the text, and generating a summary.

Abstractive Summarization
Abstractive summarizers generate summarizations consisting of novel sentences that were not part of the original text. Abstractive summarization algorithms are usually more complex because they have to understand the input text, find the most relevant passages, and generate syntactically correct sentences as summarization. Such a task is nearly impossible for hand-written rules. However, the recent advance of machine learning and, in particular, neural networks makes abstractive summarization possible. Moreover, neural networks represent the current state-of-the-art in abstractive summarization. Nallapati et al. (2016) models abstractive text summarization using Attentional Encoder-Decoder Recurrent Neural Networks. They propose several novel models that address critical problems in summarization that are not adequately modeled by the basic architecture, such as modeling keywords, capturing the hierarchy of sentence-to-word structure, and emitting rare or unseen words during the training time. Liu et al. (2017) propose an adversarial process for abstractive text summarization. Yao et al. (2018) propose a recurrent neural network-based Seq2Seq attentional model with a dual encoder including the primary and the secondary encoders. Song et al. (2019) propose an LSTM-CNN based approach that can construct new sentences by exploring more fine-grained fragments than sentences, namely, semantic phrases. The proposed approach is composed of two main stages. The first stage extracts phrases from source sentences. The second stage generates text summaries using deep learning. Liu and Lapata (2019) apply BERT in text summarization and propose a general framework for both extractive and abstractive models. For abstractive summarization, they propose a new fine-tuning schedule that adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two as the former is pretrained while the latter is not.
To the best of our knowledge, there is no work exploring the influence of named entities on the extractive summarization techniques, let alone in the Czech language.

Dataset
We use SumeCzech for experiments. SumeCzech is Czech news-based summarization dataset created by Straka et al. (2018). It contains more than a million documents, consisting of a headline, several sentences long abstract, and a full text. The dataset was collected from various Czech news websites. We show the distribution of the websites in Table 1.
SumeCzech is split into four parts. Three of them are the train, development, and test sets. Additionally, to simulate a real-life situation where a model is trained on data from one domain, and used on real data from other domains, Straka et al. (2018) created an out-of-domain (OOD) test set. OOD test set evaluates how models cope with news articles from domain never seen during training. They clustered the

Named Entity Annotations
We train a model for named entity recognition in the Czech language to annotate SumeCzech by named entities. We selected the CoNLL-based extended CNEC 2.0 (Konkol et al., 2014) as the training dataset, as it is the largest and most up-to-date Czech named entity recognition dataset. The advantage is that the dataset contains no nested entities, making the outputs easier to use for summarizers.
We selected SpaCy's NER model (Honnibal et al., 2020) because previous experiments by Müller (2020a) showed that SpaCy's NER model offers a good trade-off between performance, speed, and memory requirements. Speed and memory requirements might seem unimportant for our experiments because we can precompute the annotations. However, for the sake of practical usage, in which the labels have to be created as soon as possible once a new document for summarization arrives, we decided to take those properties into account too. The SpaCy's NER model achieved a 78.45 F-Score on the testing set of CoNLL-based extended CNEC 2.0. For comparsion, current state-of-the-art result on this dataset is 86.39 F-Score (Straková et al., 2019;Müller, 2020b).
We applied the trained SpaCy's NER model to the text of SumeCzech's articles. The result was annotations in IOB2 format, one label for each word token. The NER found approximately 26M named entities in texts, 1M in headlines, and 3M in abstracts. ( Table 2, Table 3, and Table 4. We also counted the number of headlines without any named entity. We show the statistic in Table 5. We published the annotations to promote replication of results and to enable further research (Marek and Müller, 2021).

Metrics
We used the ROUGE RAW metric for evaluation. ROUGE RAW was proposed by Straka et al. (2018) as a language-agnostic variant of ROGUE (Lin, 2004). The original ROGUE metric automatically determines the quality of the generated summary by comparing it to a reference summary created by humans. There are two variants. ROUGE-N measures the overlap of N-grams between the generated and reference summaries. ROUGE-L looks at the longest common subsequence between the reference and the generated summaries. ROGUE calculates recall and is English-specific. It employs English stemmer, stop words, and synonyms. ROUGE RAW does not need any stemmer, stop words, or synonyms, which makes it language independent. It measures recall, precision and F-score. It also has two variants ROUGE RAW -N and ROUGE RAW -L corresponding to the variants of the original ROGUE metric. We selected ROUGE RAW -1, ROUGE RAW -2, and ROUGE RAW -L to evaluate approaches that we propose.

ROUGE NE
Since we focus on the role of named entities in summarization, we propose a novel metric ROUGE NE . ROUGE NE measures the overlap of named entities between the reference and the generated summaries. This metric evaluates the ability of the model to transfer named entities to the summary.
Formally, let us denote the tokens of a true summary X: where x i are individual tokens of the summary. Let us denote the generated summary Y in a similar fashion: Y = {y 0 , y 1 , y 2 , . . . , y m }.
Next, we apply the named entity recognition algorithm on X and Y. The result is entity annotations xe i and ye i for all tokens x i and y i : The annotations can be divided into a set of annotations E NE , that mark entities, and annotations E ¬NE , that do not mark entities. In the analogy of IOB format, the former are I and B annotations, and the latter are O annotations. For the calculation of ROUGE NE we select only tokens that are marked as entities. Formally, we select only the tokens of summaries X and Y, for which its entity label is an element of E NE .
The results are the X e and Y e : Next, we calculate the ROGUE precision and recall scores using the tokens of X e and Y e as follows: where |X e | and |Y e | denote the sizes of X e and Y e . |X e ∩ Y e | denotes the number of overlapping tokens between X e and Y e . The resulting values are the precision and recall of ROUGE NE . We define the metrics to be equal to zero for summaries without any named entity.

Methods
The task we study is to create a one-sentence summary from the text of the article. The one-sentence summarization can be seen as the task to create a headline of the article. We use five baselines introduced by Straka et al. (2018). Moreover, we propose one extractive method -Named Entity Density and two abstractive approaches, Seq2Seq and Seq2Seq-NER, for text summarization.

Baselines
We adopt the methods proposed by Straka et al. (2018) as a baseline. They propose four extractive and one abstractive methods for SumeCzech's task to create a headline out of the text of the article: • First: unsupervised extractive method. It returns the first sentence of the article.
• Random: unsupervised extractive method. It returns a random sentence from the article. • TextRank: unsupervised extractive method. It selects the most important sentence of the article using the TextRank (Mihalcea and Tarau, 2004) algorithm. • clf-rf : supervised extractive method. It selects the sentence that receives the highest score produced by the Random forest classifier. The classifier performs classification using vector representation of sentences. The vector representation consists of the sum of TF-IDF for each word normalized by the sentence length, length of the sentence, cohesion (distance from other sentences), the count of capitalized words in the sentence, the count of tokens that consist of digits, and the count of non-essential words that suggests the sentence relates to some other sentence. • t2t: supervised abstractive method. It uses a neural machine translation model proposed by Vaswani et al. (2017) to generate a summary consisting of a novel sentence.

Named Entity Density
Named Entity Density is our proposed unsupervised extractive method. It calculates the named entity density score for each sentence and selects the sentence with the highest score. The score is calculated in two steps. First, we apply a named entity recognition algorithm to all sentences. Next, we calculate the named entity density as a ratio of the number of tokens marked as a named entity to the total number of tokens in the sentence.
Formally, let us denote the article A, for which we want to create a summary, as a set of sentences s 0 . . . s n : A = {s 0 , s 1 , s 2 , . . . , s n }.
Each sentence s i contains word tokens x 0 . . . x m : We apply the named entity recognition algorithm NER on each sentence s i of the where |E NE | denotes the number of tokens in the sentence s i which NER algorithm marked as named entities and |s i | is the number of all tokens x 0 . . . x m forming the sentence s i . We select the sentence s i with the highest NED score as a summary of article A.
The intuition of the Named Entity Density is that the sentence with the high NED score mentions the highest number of entities within the smallest text fragment. Such a sentence corresponds to the form of a report that is structured around concisely identifying and examining issues, events, or findings that have happened.

Seq2Seq
Seq2Seq is a supervised abstractive method that uses a Seq2Seq model with global attention. Formally, a Seq2Seq neural network models the conditional probability p(y|x) of translating a source text x = {x 0 , x 1 , x 2 , ..., x n } into a target text y = {y 0 , y 1 , y 2 , ..., y m } (Luong et al., 2015). The source text x is an article, y is a summary, and m < n in our case. The Seq2Seq neural network consists of an encoder and a decoder. The encoder creates a fixed-length vector representation r of the source text x: r = ENC(x).
The encoder is usually a recurrent neural network with hidden states h ENC s : and the output of the encoder is its last hidden state: The f ENC can be a vanila RNN (Rumelhart et al., 1985), LSTM (Hochreiter and Schmidhuber, 1997), or GRU (Cho et al., 2014) unit. The decoder takes r as an input and generates a target text, one token at a time: log p(y|x) = m j=1 log p(y j |y <j , r).
We can also represent the probability of generating a target word y j as: where g is a transformation function that generates vocabulary sized vector. The h DEC j is the output of recurrent neural network unit: Function f DEC can be a vanilla RNN, LSTM, or GRU unit like in the case of the encoder. We add a global attention mechanism to the Seq2Seq neural network. The attention allows the network to focus on parts of the source text selectively during the target text generation. We illustrate the global attention mechanism in Figure 1.

Attention Layer
Context vector Figure 1. Seq2Seq with global attention mechanism. The figure is inspired by Luong et al. (2015).
The idea is to concatenate a source-side context vector c j with the input vector of the encoder y j−1 :ỹ The context vector c j is computed as a weighted average over all vectors of hidden states of the encoder h ENC s : where a j i is the i th element of the weight vector a j . The a j is calculated by the softmax function comparing each hidden state of the encoder h ENC s with the current hidden state of the decoder h ENC j : ) .
There are multiple definitions of the score function. We selected the general score function, defined as: score(h t , h s ) = h t Wh s .

Seq2Seq-NER
Seq2Seq-NER is a supervised abstractive method that uses a Seq2Seq model with global attention and adds the NER feature encoded as one-hot encoded vector appended to input embedding vector. Formally, the Seq2Seq neural network models the conditional probability p(y|x NER ), where y is a target text y = {y 0 , y 1 , y 2 , ..., y m } and x NER is a source sequence. The source sequence is a concatenation of a vector representation of the source token x i and one-hot vector representation of entity type e i : x The rest of the model works in a similar fashion as a Seq2Seq model, that we described in the subsection 5.3. To summarize, the difference between Seq2Seq and Seq2Seq-NER models is that the latter has access to the named entity labels of the source words produced by the NER algorithm.

Implementation Details
We implemented the baseline methods first and random proposed by Straka et al. (2018). We replicated results using ROUGE RAW -1, ROUGE RAW -2, and ROUGE RAW -L metrics, and additionally evaluated ROUGE NE metric.
For the proposed supervised methods, we used a modified implementation of a Seq2Seq model with global attention from the official PyTorch tutorial (Weidman, 2019). The hidden sizes of the encoder and decoder were set to 256. The size of our vocabulary was 25,000. We used 300-dimensional fastText for embedding words. We used dropout 0.1 on the outputs of both RNNs. We trained our models until the validation loss started increasing, and we selected the weights having the lowest validation loss for evaluation.
We encountered problems with attention. The implementation in the tutorial used attention similar to concat global attention. The tutorial suggested to copy hidden state of the decoder for each hidden state of the encoder, then concatenate the hidden states with the encoder outputs, and pass them to a linear layer to calculate energy. The reason was to utilize parallel nature of GPU. We used batch size 16. Therefore, the concatenated tensor would have 147,890,688 floating numbers in the case of the longest sentence. To train our model with attention without running out of memory on our GPU, we had to simplify the way attention was calculated. We used an approach similar to general global attention. We used an affine transformation on the hidden state of the decoder to transform it into a 64-dimensional vector to calculate energy. We also affinely transformed the encoder's hidden states vectors of the same dimension by a linear layer. The transformed decoder hidden state was then used as a multiplier and broadcast over all encoder hidden states, making the calculation of energy much more memory efficient because the hidden state of the decoder did not have to be copied.
A limited vocabulary of the model led to many unknown words in the titles, and the model that used word tokenization learned to predict them. We had to forbid the model from predicting unknown tokens during evaluation to get meaningful titles.
For the Seq2Seq-NER model, we encoded the entities using the IOB2 format. The format has one common outside tag, and beginning and inside tags for each entity type. We encoded NER features into a one-hot vector for each word. The vector has 17 dimensions. Fourteen dimensions are reserved for the beginning and inside tags for each of the seven entity types our NER distinguishes. One dimension represents the outside of the entity tag. One represents padding, and one represents both the start and end of sequence symbols. We concatenated the NER feature vector with the embedding vector entering the encoder of the Seq2Seq.

Results of Automatic Evaluation and Discussion
We show the results of the evaluation in Table 6. First, we replicated the results of First and Random baselines reported in Straka et al. (2018). Our results were on par with the reported Precision, Recall, and F-Score of Rogue Raw -1, Rogue Raw -2, Rogue Raw -L. We additionally evaluated our proposed metric Rogue NE for comparison with other methods.
Next, we evaluated the proposed extractive method Named Entity Density (NE Density). Results of Named Entity Density compared to the First, a solid summarization baseline, especially in the news articles domain, are encouraging. Named Entity Density achieves only slightly worse results. Moreover, the achieved results are consistent between the test and the OOD test set. Additionally, as we will show in section 8, Named Entity Density produces summaries resembling the style of informationally concise reports.
We evaluated the Seq2Seq and Seq2Seq-NER on the Test set to compare those methods with the approaches proposed by Straka et al. (2018). We can see that our proposed Seq2Seq and Seq2Seq-NER methods achieve better results on average by 80% relatively in Precision and F-score compared to the best methods proposed by Straka et al. (2018). Only Textrank and First achieve better results in Recall. The Seq2Seq-NER achieved slightly better results than Seq2Seq, which proves NER labels' usefulness for summarization. Although, it seems from the results of Rogue NE that the better score is not caused by the improved performance of using entities in the summaries.
We evaluated the Seq2Seq and Seq2Seq-NER on the OOD test set to learn how models cope with news articles from a domain never seen during training. The results are encouraging. Even though they show a drop in absolute values of metrics between Test and OOD test sets, the trend is the same. Seq2Seq and Seq2Seq-NER methods achieve the best results of all compared methods in Precision and F-score, and Seq2Seq-NER has slightly better results than Seq2Seq.
Finally, we take a look at the results in Rogue NE . We do not have results for Textrank and Tensor2Tensor because they were not reported in the work of Straka et al. (2018) and we did not implement the methods ourselves. However, it is clear from the rest of the results that even the recent state-of-the-art methods are struggling with the named entities in the summarization.  16.2 14.1 14.7 2.5 2.1 2.2 14.7 12.8 13.3 NER 13.7 11.9 12.4 2.0 1.7 1.8 12.6 11.1 11.4 0.9 0.9 0.9 Table 6. Results of automatic evaluation

Examples
We choose a few representative examples from the test and OOD test sets to show how different methods summarize. We also provide English translation for convenience. Only very simple automatic post-processing was done on the output of the proposed Seq2Seq and Seq2Seq-NER models. We filtered the start of the sentence   Table 7. We do not divide the examples into test and OOD test sets because the generated summaries of both sets achieve comparative quality thanks to the fact that Named Entity Density is an unsupervised method.
We can see that the selected sentences contain named entities. Those sentences are comprised of factual information. The sentences are always grammatically correct thanks to the fact that Named Entity Density is an extractive approach. Even though the summaries created by First can contain more entities in general, the summaries created by Named Entity Density have a higher density of entities. We can see that the summaries created by Named Entity Density revolve around to whom, when, where, and what happened. It closely resembles the style of reports that concisely identify and examine issues, events, or findings that have happened.
Notice also that the sentences selected by Named Entity Density are not the first sentences of the articles. We measured that the sentence selected as a summary by Named Entity Density differs from the sentence selected by First in 93% of SumeCzech's articles. Thus, we can use the summaries created by Named Entity Density as an alternative version or reformulation of summaries created by method selecting the First sentence of an article. This property is highly praised by voice applications like Alquist (Pichl et al., 2020) or Emora (Finch et al., 2020). Voice applications present news articles in a summary because users quickly lose focus as news articles are not intended to be read by synthetic voices. Initially, the voice application can present the first sentence of the article. Additionally, if the user requests to learn more, it can present the summary produced by Named Entity Density.
We show the results of Seq2Seq and Seq2Seq-NER models for test and OOD test sets separately in Table 8 and Table 9. Both models generate novel sentences and incorporate entities into the generated summarizations successfully. We can see that despite the promising results of the automatic evaluation, a part of the outputs are not grammatically correct and contain repeated words.

Conclusion
This work explored the summarization of Czech news articles and influence of named entity labels for this task. We selected the SumeCzech dataset for our experiments. SumeCzech is over one million articles large dataset collected by Straka et al. (2018) from Czech news websites. We annotated SumeCzech by named entities by the SpaCy's NER. We published the annotations to promote replication of the results and to enable further research.
We used the methods introduced by Straka et al. (2018) as a baselines, namely First, Random, TextRank, clf-rf, and t2t. We selected a task to create a headline out of the text of the article, which can be considered as a single sentence summary.  We proposed an extractive approach Named Entity Density that selects a sentence with the highest ratio between the number of entities and length of the sentence as the summary of the article. The experiments showed that Named Entity Density achieved nearly as good results as baseline selecting the first sentence of the article, which is a very hard baseline to beat, especially in the domain of news articles. Nevertheless, the summaries generated by Named Entity Density demonstrated that the selected sentences reflect the style of reports concisely identifying to whom, when, where, and what happened. We proposed using a combination of Named Entity Density and First summaries in voice applications. The voice application can initially present the first sentence of the article, and continue by follow-up created by Named Entity Density if a user requests additional information.
Next, we proposed two abstractive approaches based on the Seq2Seq architecture. The first approach, Seq2Seq, generates novel summaries using only tokens of the article's text. The second approach, Seq2Seq-NER, additionally uses the named entity labels of each word as its input. Experiments showed that both proposed methods achieve better results than the methods proposed previously by Straka et al. (2018). Seq2Seq-NER improved the results over Seq2Seq in automatic evaluation. This result demonstrated the usefulness of named entity labels for summarization. Furthermore, the results of the methods showed similar trends even on the out-of-domain test set.  Finally, we proposed a new metric, ROGUE NE , that measures the overlap of named entities between the true and generated summaries. The results show that the current state-of-the-art methods struggle with named entities in summarization, and there is a significant opportunity for further research.