Word embedding models are typically able to capture the semantics of words via the distributional hypothesis , but fail to capture the numerical properties of numbers that appear in a text . This leads to problems with numerical reasoning involving tasks such as question answering . We propose a new methodology to assign and learn embeddings for numbers . Our approach creates Deterministic , Independentof-Corpus Embeddings ( referred to as DICE ) for numbers , such that their cosine similarity reflects the actual distance on the number line . DICE outperforms a wide range of pre-trained word embedding models across multiple examples of two tasks : ( i ) evaluating the ability to capture numeration and magnitude ; and ( ii ) to perform list maximum , decoding , and addition . We further explore the utility of these embeddings in downstream applications by initializing numbers with our approach for the task of magnitude prediction . We also introduce a regularization approach to learn model-based embeddings of numbers in a contextual setting . Word embeddings capture semantic relationships between words by operationalizing the distributional hypothesis ( Harris , 1954 ; Firth , 1957 ) . They can be learned either non-contextually ( Mikolov et al . , 2013b ; Pennington et al . , 2014 ; Bojanowski et al . , 2017 ) or contextually ( Devlin et al . , 2018 ; Peters et al . , 2018 ) . Non-contextual embeddings have worked well on various language understanding and semantic tasks ( Rumelhart et al . , 1988 ; Mikolov et al . , 2013a , b ) . More recently , they have also been used as pre-trained word embeddings to aid more sophisticated contextual models for solving rigorous natural language processing ( NLP ) problems , including translation , paraphrasing , and sentence-similarity tasks ( Kiros et al . , 2015 ; Wieting et al . , 2015 ) . While word embeddings effectively capture semantic relationships between words , they are less effective at capturing numeric properties associated with numbers . Though numbers represent a significant percentage of tokens in a corpus , they are often overlooked . In non-contextual word embedding models , they are treated like any other word , which leads to misinterpretation . For instance , they exhibit unintuitive similarities with other words and do not contain strong prior information about the magnitude of the number they encode . In sentence similarity and reasoning tasks , failure to handle numbers causes as much as 29 % of contradictions ( De Marneffe et al . , 2008 ) . In other data-intensive tasks where numbers are abundant , like neural machine translation , they are masked to hide the translation models inefficiency in dealing with them ( Mitchell and Lapata , 2009 ) . There are a variety of tests proposed to measure the efficiency of number embeddings . For instance , Naik et al . ( 2019 ) shows that GloVe ( Pennington et al . , 2014 ) , word2vec ( Mikolov et al . , 2013b ) , and fastText ( Joulin et al . , 2016 ; Bojanowski et al . , 2017 ) fail to capture numeration and magnitude properties of a number . Numeration is the property of associating numbers with their corresponding word representations ( " 3 " and " three " ) while magnitude represents a number 's actual value ( 3 < 4 ) . Further , Wallace et al . ( 2019 ) proposes several tests for analyzing numerical reasoning of number embeddings that include list maximum , decoding , and addition . In this paper , we experimentally demonstrate that if the cosine similarity between word embeddings of two numbers reflects their actual distance on the number line , the resultant word embeddings are useful in downstream tasks . We first demonstrate how Deterministic , Independent-of-Corpus Embeddings ( DICE ) can be constructed such that they almost perfectly capture properties of numera-tion and magnitude . These non-contextual embeddings also perform well on related tests for numeracy ( Wallace et al . , 2019 ) . To demonstrate the efficacy of DICE for downstream tasks , we explore its utility in two experiments . First , we design a DICE embedding initialized Bi-LSTM network to classify the magnitude of masked numbers in the 600 K dataset ( Chen et al . , 2019 ) . Second , given the popularity of modern contextual model-based embeddings , we devise a regularization procedure that emulates the hypothesis proposed by DICE and can be employed in any task-based fine-tuning process . We demonstrate that adding such regularization helps the model internalize notions of numeracy while learning task-based contextual embeddings for the numbers present in the text . We find promising results in a numerical reasoning task that involves numerical question answering based on a sub-split of the popular SQuAD dataset ( Rajpurkar et al . , 2016 ) . Our contribution can be summarized as follows : • We propose a deterministic technique to learn numerical embeddings . DICE embeddings are learned independently of corpus and effectively capture properties of numeracy . • We prove experimentally that the resultant embeddings learned using the above methods improve a model 's ability to reason about numbers in a variety of tasks , including numeration , magnitude , list maximum , decoding , and addition . • We also demonstrate that properties of DICE can be adapted to contextual models , like BERT ( Devlin et al . , 2018 ) , through a novel regularization technique for solving tasks involving numerical reasoning . In this work , we methodologically assign and learn embeddings for numbers to reflect their numerical properties . We validate our proposed approach with several experiments that test number embeddings . The tests that evaluate the numeral embeddings are fundamentally applicable to all real numbers . Finally , we introduced an approach to jointly learn embeddings of numbers and words that preserve numerical properties and evaluated them on a contextual word embedding based model . In our future work , we would like to extend this idea to unseen numbers in vocabulary as a function of seen ones .
Existing word embeddings treat numbers like words failing to capture numeration and magnitude properties of numbers which is problematic for tasks such as question answering.
They propose a deterministic technique to learn numerical embeddings where cosine similarity reflects the actual distance and a regularization approach for a contextual setting.
A Bi-LSTM network initialized with the proposed embedding shows the ability to capture numeration and magnitude and to perform list maximum, decoding, and addition.