Metaphor Detection using Deep Contextualized Word Embeddings

Metaphors are ubiquitous in natural language, and their detection plays an essential role in many natural language processing tasks, such as language understanding, sentiment analysis, etc. Most existing approaches for metaphor detection rely on complex, hand-crafted and fine-tuned feature pipelines, which greatly limit their applicability. In this work, we present an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs and multi-head attention mechanism to address the task of automatic metaphor detection. Our method, unlike many other existing approaches, requires only the raw text sequences as input features to detect the metaphoricity of a phrase. We compare the performance of our method against the existing baselines on two benchmark datasets, TroFi, and MOH-X respectively. Experimental evaluations confirm the effectiveness of our approach.


Introduction
A metaphor is a figurative form of expression that compares a word or a phrase to an object or an action to which it is not literally applicable but helps explain an idea or suggest a likeness or analogy between them. Metaphors have been used extensively in all types of literature and writings, especially in poetry and songs to communicate complex feelings, emotions, and visuals present in the text to readers effectively. Metaphors are ubiquitous in natural language and help in structuring our understanding of the world even without our conscious realization of its presence [1]. Given the prevalence and significance of metaphorical language, effective detection of metaphors plays an essential role in many natural language processing applications, for example, language understanding, information extraction, sentiment analysis, etc.
Email addresses: shashwata.co@nsit.net.in (Shashwat Aggarwal), rsingh@nic.in (Ramesh Singh) However, automated detection of metaphorical phrases is a difficult problem primarily due to three reasons. First, there is a subjective component involved: the metaphoricity of expression may vary across humans. Second, metaphors can be domain and context dependent. And third, there is a lack of annotated data, which is required to train supervised machine learning algorithms to facilitate automated detection accurately.
Most of the previous approaches for detection of metaphorical phrases, have either relied on manual and lexical detection [2,3,4] which requires heavily handcrafted features built from linguistic resources, that are costly to obtain and greatly limits their applicability or have used supervised machine learning based algorithms [5,6,7] with limited forms of linguistic context, for example using only the subject verb objects triplets (e.g. cat eat fish). Although these techniques automate the detection of metaphorical phrases, however, the prediction accuracies are not as good as the prediction accuracies of these techniques in other text classification tasks.
Inspired by recent works in the field of NLP and transfer learning, in this paper, we present an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs and multi-head attention mechanism to address some of the limitations aforementioned. Our method is notable in the sense that unlike many existing approaches, it requires only the raw text sequences as input and does not depend on any complex or fine-tuned feature pipelines.

Literature Survey
There has been significant work done in automatic detection and discovery of metaphorical phrases in natural language ranging from the traditional rule-based methods which rely on task-specific hand-coded lexical resources to the recent statistical machine learning models to identify metaphors.
One of the first attempts to detect metaphorical phrases automatically in the text was by [8]. Their system called met* could discriminate between literalness, metonymy, and metaphor in the underlying text. [9] proposed a sentence clustering approach for non-literal language recognition using a similarity-based word sense disambiguation method. However, their approach focused only on metaphors expressed by a verb. [10] trained a maximum entropy classifier to discriminate between literal and metaphorical usage of a phrase. Their method got a high accuracy of 95.12%. However, the majority of the verbs present in their dataset were used metaphorically, thus making the task notably easier. [3] used hyponymy (IS-A) relation in WordNet [11] and word bigram counts to predict metaphors in sentences. However, the word bigram counts lose a great deal of information over the verb-noun pairs. Also, they do not deal with literal sentences.
[12] used statistical learning methods to identify metaphors automatically. They start with a small seed of manually annotated metaphorical expressions. The system generates a large number of metaphors of similar syntactic structure from a corpus.
Other approaches using statistical learning methods to identify metaphors automatically include [4], that use concreteness and abstractness to detect metaphors, [13], that apply a Latent Dirichlet Allocation (LDA) based topic modeling method for automatic extraction of linguistic metaphors, [14] that construct a metaphor detection system using a random forest classifier with conceptual semantic features such as abstractness, imageability, and semantic supersenses, etc. Most of these existing approaches use a variety of features which rely on external lexical, syntactic, or semantic linguistic resources to train classification models for metaphor detection which severely limits their applicability.
Recently, there has been an introduction of several deep learning based approaches to tackle the problem of metaphor detection. Several methods such as [15,16,7] train word embeddings to identify and detect metaphors and have shown gains on various benchmarks. [17] used bidirectional LSTMs with raw text, SVO and dependency subsequences as input features. The 2018 VUA Metaphor Detection Shared Task has also introduced several LSTM and CRF based models [18,19,20,21] which are augmented by linguistic features such as POS tags, lemmas, verb clusters, unigrams, WordNet, etc.
Furthermore, there have been works such as [22,23] which have employed the recently proposed contextualized language representation models to detect metaphors. However, the work in this direction is limited. Also, a recently proposed popular multihead attention mechanism [24], which has been used extensively in domains such as speech processing or machine translation in the recent past has also not been tried for the task of metaphor detection yet.
Based on the lines of some of these recent works, in this paper, we combine the contextualized language representation models, ELMo [25] and BERT [26] with multihead attention mechanism and bidirectional LSTMs to detect metaphorical phrases automatically. We compare our proposed approach with existing baselines for the task of metaphor detection and classification.
The rest of the paper is organized as follows: In section 3, we discuss our proposed approach and in section 4 & 5, we describe the datasets we use and the experiments we perform on them.

Proposed Approach
We present an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs and multi-head attention mechanism to address the task of automatic metaphor detection and classification. Our method takes the raw text sequence as input and does not depend on any complex, hand-crafted or fine-tuned feature pipelines.

Contextualized Word Representations
Natural language processing (NLP) is a diversified field consisting of a wide variety of tasks such as text classification, named entity recognition, question answering, etc. However, most of these tasks contain limited datasets with a few thousand or a few hundred thousand human-labeled training examples. This shortage of training data severely affects the performance of modern deep learning-based NLP systems, which proffer benefits when trained from much larger amounts of data.
Recently, there have been techniques such as [25,26], that address this limitation by pretraining language representation models on enormous amount of unannotated text and then transferring the learned representation to a downstream model of interest as contextualized word embeddings. These contextualized word representations have resulted in substantial gains in accuracy improvements in numerous NLP tasks as compared to training on these tasks from scratch. Following recent work, we also use these contextualized word embeddings for our task of metaphor detection. Figure 1 shows the architecture of our proposed method for the task of metaphor classification. Each token t i of the input text is encoded by the contextualized word embeddings, ELMo or BERT, to obtain the corresponding contextualized word representations. The contextualized representations are feed as input to the bidirectional LSTM encoder network. The encoder network outputs an encoded representation across every timestep of the input sequence of text.

Proposed Model
We add a multi-head attention mechanism on top of the encoder network to obtain a context vector c representing the weighted sum of all the BiLSTM output states as depicted in equation 1, where h is the number of attention heads used, x i corresponds to the hidden representations from the encoder network, a j i is the attention weight computed for token t i from attention head j and c j is the context vector obtained from attention head j respectively.
Finally, the context vector c is feed to a dense layer decoder to predict the metaphoricity of the input phrase.

Datasets
We evaluate our method on two benchmark datasets, TroFi [9] and MOH-X [6] respectively. In Table 1, we report a summary of each of the dataset used in our study. The TroFi dataset consists of literal and nonliteral usage of 50 English verbs drawn from the 1987-89 Wall Street Journal (WSJ) Corpus. The MOH-X dataset, on the other hand, is an adaptation of the MOH dataset [27] which consists of simple and concise sentences derived from various news articles.

Implementation Details
We use 1024d ELMo and BERT vectors. The LSTM module has a 256d hidden state. We use 4 attention heads for the multi-head attention mechanism. We train the network using Adam optimizer with learning rate lr = 0.00003, β1 = 0.9, β2 = 0.999 and with a batch size of 16. The network is trained for 50 epochs.

Baselines
We compare the performance of our method with four existing baselines. The first baseline is a simple lexical baseline that classifies a phrase or token as metaphorical if that phrase or token is annotated metaphorically more frequently than as literally in the training set. The other baselines include a logistic regression classifier similar to one employed by [28], a neural similarity network with skip-gram word embeddings [7], and a BiLSTM based model combined with ELMo embeddings proposed in [22].

Evaluation Metrics
To evaluate the performance of our method with the baselines, we compute the precision, recall, F1 measure for the metaphor class and the overall accuracy. We perform 10 fold cross-validation following prior work.

Results
In Table 2, we show the performance of all the baselines and our proposed method on the task of metaphor detection and classification for two benchmark datasets (MOH-X and TroFi). Both of our proposed methods outperform the existing baselines on both the datasets using only the raw text as input features. The model with BERT performs better as compared to the model with ELMo embeddings on both the datasets. Contextualized word embeddings and multi-head attention mechanism improve the performance of both models by a large margin.
In addition to the performance comparison of our methods with the existing baselines, we also show the receiver operating characteristic curve in Figure 2 showing the AUC values for the metaphor detection task and illustrating the diagnostic ability of the best method (ours w/ BERT) as its discrimination threshold is varied.
Further, In Table 3 & 4, we report a set of examples from the TroFi and MOH-X developement sets along with the gold annotation label, the predicted label from both of our methods (with ELMo and with BERT) and the corresponding prediction scores assigned. The system confidently detects most of the metaporical and literal phrases such as "make the people's hearts glow.", "he pasted his opponent." and respectively.  However, the predicted output disagrees with the ground truth annotations for some phrases such as "the white house sits on pennsylvania avenue." and "bicycle suffered major damage." respectively. Majority of errors committed by our method are in detection of a particular type of metaphor called personification. The metaphoricity of phrases such as "the white house sits on pennsylvania avenue." are largely dependant on a few words like "white house" which if replaced by the name of some person changes the metaphoricity of the phrase. The network occasionally gets confused in discriminating between different entities like people, things, etc. Given additional information along with raw text as input to the network, for example the part-of-speech or named-entity tags, the performance of the network could be improved.
Finally, we report some example phrases such as "vesuvius erupts once in a while." or "the old man was sweeping the floor." for which the model with BERT embeddings correctly detects the metaphoricity of the phrase, while the model with ELMo fails to do so. Table 3: Examples from the TroFi development set, along with the gold label, predicted label, and the predicted score from our method with ELMo and BERT.

Input Phrase
Gold Predicted Score (for Metaphor Class) w/ ELMo w/ BERT w/ ElMo w/ BERT "make the people's hearts glow ."  Table 4: Examples from the MOH-X development set, along with the gold label, predicted label, and the predicted score from our method with ELMo and BERT.
Input Phrase Gold Predicted Score (for Metaphor Class) w/ ELMo w/ BERT w/ ElMo w/ BERT "this one speech could sink his candidacy."

Conclusion
In this work, we presented an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs, and multi-head attention mechanism to address the task of automatic metaphor detection and classification. Our method requires only the raw text sequences as input and does not depend on any complex or fine-tuned feature pipelines. Our method established new state-of-the-art on both the datasets for metaphor detection.