Physics Computer Science
Computer Science-2021.gz:143251
Computer Science
97cb0160ea72a63fe68c2f924c7578248156ae21
An adaptive approach for word sense disambiguation for Hindi language

Word sense disambiguation has been a classical problem and it is still considered as a big challenge for NLP researchers. Word vectors have shown remarkable improvements from baseline performances for various NLP challenges. An adaptive approach is developed to resolve the ambiguity at word level using word vectors and presented in this paper. CBOW and Skip-gram models of Word2Vec are used to generate the word embeddings. An input sentence is split into two sets: The first set contains all possible ambiguous words and the second set is constituted from all the unambiguous words. The words from the first set are processed through IndoWordNet to collect all the senses of respective words, and such senses are stored in sensedictionary. An adaptive approach is applied to find out the exact interpretation of the ambiguous word from the sense dictionary for each word from the set of ambiguous words with the context words from the set of unambiguous words. The proposed approach is evaluated on a large scale Hindi corpus and claimed better results than previous attempts.


Introduction
Natural Languages are ambiguous by default. A sentence can be ambiguous due to one or many ambiguous words or due to the order of the words. Sometimes the punctuation also plays a vital role in multiple meaning of a sentence. Word Sense Disambiguation (WSD) is used to find out the correct and relevant meaning (sense) of a word which can have multiple meaning in the sentence. Without applying Word Sense Disambiguation, a sentence can be interpreted in more than one meaning. Out of these multiple meanings of the sentence, at least one of the meanings will be always correct however sometimes more than one meaning are also sensible. The WSD is counted in initial steps of language processing, as without it, sense of a sentence can be misinterpreted [1]. In normal human communication, humans perceive secondary information like context from the environment and the sender of the information. With the support of such information, humans can perceive the right meaning of a word out of multiple meanings easily. However such crucial information is not available in the text while processing it by machines which results to develop the correct meaning of the sentence is a challenging task. WSD is defined as the "ability to identify the meaning of words in context in a computational manner" by Navigli [3]. A word may act ambiguously at any place in a sentence or the word may act ambiguously in a specific role or position in the sentence. In the first case, it is required to perform WSD on all words in a sentence however in the later one WSD is applied only on the specific word. Based on this, there are two classifications-WSD for all words in a sentence and WSD for Lexical-sample. Researchers are taking advantage of huge web crawled data in the form of corpus and applying different deep/shallow learning methods to generate word embeddings or word vectors. A word embedding or IOP Publishing doi: 10.1088/1757-899X/1131/1/012022 2 word vector is a learned representation for words where the words with similar semantics have related and close representation. In humans, the meaning of a word is associated with certain neurons and weights among their interconnection in the brain. Similar words with related meaning form logical groups associated together. The word vectors also follow the same principle. The dimensions of such vectors vary from 50-600 as per the requirement of the application and availability of computing resources. These word vectors are represented using real numbers to imitate the behaviour of a word and have performed extremely well in various NLP tasks. There are various approaches to generate word embeddings like Hard and Soft Clustering models, Mathematics based counting models and Neural-Network based models. The word vectors generated using neural-network based models specifically word2vec and fastText outperformed vectors generated by other models in most of the NLP tasks. Most of the research works have been carried out in the English language for the generation of word vectors for the English Language. Researchers have focused more on certain languages like Arabic, Chinese, French Japanese and Russian. However, few attempts like Polyglot have been made to develop word vectors in a group for multiple languages in one go. For the Hindi Language too, continuous efforts are in progress to develop quality word vectors. This paper is an attempt to create word embeddings using word2vec and use these word embeddings for word sense disambiguation. The next section presents an extensive survey of the work done in the domains of WSD and Word Representation respectively. Section III describes the proposed work and after that, the methodology is explained in Section IV. Experimental results are discussed in Section V. The last section covers the conclusion and future work in a nutshell.

Word Sense Disambiguation:
The automatic WSD is considered as a classical problem in NLP domain from the starting of automated language processing. With the genesis of automatic machine translation, WSD was assumed as a subtask. Numerous research attempts have been made for the English WSD, however, for the Hindi language, WSD challenge is still a hot topic. Ide et al. discussed WSD in detail and termed it as "AI-Complete". That is, "a problem which can be solved only by first resolving all the difficult problems in artificial intelligence (AI), such as the representation of common sense and encyclopedic knowledge" [1]. Agirre and Stevenson [2] provided a thorough analysis of finding clues through the syntactic and semantic analysis of a sentence in 2007. Navigli relied on knowledge-based approaches and also mentioned that supervised methods outperform unsupervised methods [3]. Zhong et al. took designed a system to take leverage of POS tagging, context window of 5 surrounding words and local collocations to solve the challenge of WSD [4]. Taghipour et al. proposed a framework using word embeddings and previous model with changing the context window to 7 surrounding words and 11 local collocations around the word [5]. Rothe and Hinrich used synsets and word embeddings to propose a model with an assumption that a word was a sum of its lexeme [6]. Chen et al. presented a unified model which developed one representation per sense and outperformed other systems [7]. They also developed domain-specific embeddings focusing on Sports and Finance domain. Research Work in for WSD in the Hindi language has not been as vast as it is in the English Language. The first attempt is considered by Sinha et al. who applied a statistical technique to disambiguate words with the help of wordnet [8] in 2004. Mishra et al. proposed a solution using a classifier which learned from unlabeled Hindi corpus [9] in 2009. Singh et al. proposed an algorithm using hypernym, hyponym, holonym and meronym relations to solve the ambiguity and proved that hyponyms could help. [10]. Agarwal and Bajpai et al. explored the disambiguation of nouns using wordnet in 2014 [11]. Patel and Bhatt proposed a solution based on Hierarchical clustering technique [12]. In 2015, Tayal et al. presented a solution using Fuzzy C-Means Clustering to create sense topics with which they resolved the ambiguity of the word [13]. In 2016, Singh et al. take a step forward in the direction to resolve ambiguity by developing annotated sense corpus [14]. In 2018, Athaiya et al. presented a genetic new IOP Publishing doi:10.1088/1757-899X/1131/1/012022 3 approach using genetic algorithm with the feature of a dynamic window subject to the condition of availability of topic in the neighbourhood of the ambiguous word [15]. Sharma and Joshi developed a system which used the knowledge-based LESK algorithm and Wordnet [16]. Kumari and Lobiyal proposed a solution to use word embeddings to resolve the ambiguity [17].

Distributed Representation of Words
The one-hot representation is one where each word is represented as a sparse vector of the same size of the vocabulary and only one value is set 1. The method of representing a word in one-hot vector representation suffers from the curse of dimensionality as the one-hot vectors are quite large yet sparse. Distributed representation of words (Word Embeddings) resolved this challenge by representing the vectors in limited dimensions with real values. Consider a vocabulary created using the following two documents: D1 and D2: D1: यह आम रास्ता नही है | D2: मु झे आम खाना पसन्द है | So after removing stop-words, the resultant Vocabulary V is. V = { 'आम', 'रास्ता', "खाना" , "पसन्द" } Examples of one-hot representation and distributed representation are mentioned in Table-1 and  Table- [19]. This model was developed using a feed-forward neural network to forecast the next word. Collobert and Weston fetch semantic information from a large corpus in the form of word embeddings in their unified architecture [20]. Mikolov et al. in 2013 brought a kind of revolution in the research world by proposing CBOW and Skipgram Models and word2vec tool [21]. These two models did not only generate effective word vectors on the basis of the semantics of the word but also were able to manipulate the meaning of the word if its vector is changed after performing vector operations on it. The CBOW takes surrounding words and forecasts the target word, on the contrary, the Skip-gram model accepts a word and predicts its neighbour/context words. Joulin et al. used sub-word information like morphological variations to generate improved vectors [23]. Different Embedding Models and their objective functions are presented and diccussed in detail in [24]. In this work, we have used variations in tuning parameters of word2vec. Three tuning parameters are considered for such variations are-1. Vector Dimensions, 2. Window Size and 3. Negative Sampling.

Proposed Approach
First, the corpus is collected using various methods and then it was pre-processed using standard methods to clean and make it ready to use. This pre-processing includes Tokenization and Normalization with the following steps: • Remove numbers and special characters • Replacing punctuation with white spaces • Stripping white spaces • Filtering stop-words Siddiqi et al. have presented an extensive list of more than 800 stop-words in their research work [25]. This list has been used in this research work to filter stop-words from the corpus as well as from the test sentence. Afterwards, the word vectors are generated using this pre-process corpus. These word vectors are later used in the second algorithm to map words from the given test sentence to their vector forms. The first algorithm WSD_VectorGeneration is shown below: A given test sentence is segregated to two sets: The first set contains all possible ambiguous words from the test sentence and the second set is constituted from all the unambiguous words. The words from the first set are processed through IndoWordNet to collect all the senses of respective words, and such senses are stored in sense-dictionary. An adaptive approach is applied to discover the exact sense of the ambiguous word from the sense dictionary for each word from the set of ambiguous words with the help of context words from the set of unambiguous words. Finally, the Cosine Similarity is used as a standard measure to evaluate the semantic relation between the word vectors.

Methodology
This system works in two phases. In the first phase, the corpus is pre-processed and Word Vectors are generated from this pre-processed corpus. The corpus used to implement this paper is formed using two different parts: first, Indian Corpora from NLTK and second, Web crawled data from Wikipedia. After forming the corpus by amalgam, it was pre-processed using the steps mentioned in the previous section. To generate word vectors from this corpus, we have to finalize the tuning parameters of CBOW and Skip-gram models. The variations in tuning parameters are arranged in three different combinations for both CBOW and Skip-gram. In the first combination, the dimensions of vectors are fixed as 50, with a window size of 15 words and no negative sampling. The second combination consists of the vector dimensions as 150, window size as 10 words and with a negative sampling of 5 words. The vector dimensions and negative sampling size are increased to 200 and 10 respectively while the window size is decreased to 5 in the last combination. 100 and 150. With this setup, the word vectors are generated. The same process is shown in Figure-  The steps of the first algorithm are self-explanatory, however, the second algorithm is explained below due to its complexity: Where CVD represents the Context Dictionary. CV1 represents the context vectors for word-1 and CVA represents the context vectors for word-A And A represents the total number of words in the dictionary.
7. Fetch sense of each word of the ambiguous set using Indowordnet. The Sense Dictionary is then updated by entering the word and its fetched (multiple) meanings from the Indowordnet.
Since Ambiguous word set has a total of "A" words so the Sense dictionary will also have total "A" entries. However, there will be multiple entries of senses against each word. 11. Generation of Sentence Decode Vectors-It is the same as generating TSV except that to create this, we put the Reference Vector instead of Vectors of ambiguous words. Since ambiguous words will have more than one meaning so this will result in a matrix of multiple vectors. This SDV Matrix, each column represents one word and one row represents a complete sentence with one meaning of the ambiguous word and thus this SDV matrix contains all the possible interpretations of the sentence which might be meaningful or meaningless.
Where m is the number of words in Test Sentence after tokenization.
12. Selection of the reference vector which is semantically most close to Vector of target Sentence can be done using 13. The sense related to the vector ( ) with a minimum cosine distance from is the best interpretation of the ambiguous word.
To explain the process further, we have taken a test sentence with an ambiguous word -"आम". The Glosses of the target word "आम", fetched from IndoWordNet are shown in Table-3.

Reference Vectors:
Reference Vectors are to be created for each sense for an ambiguous word. There are 9 Synsets mentioned for the ambiguous word -"आम", however only 8 glosses/senses are available, therefore only 8 reference vectors are generated.
Now after obtaining these Reference Vectors for the ambiguous word, the Sentence Decode Vectors are computed as mentioned below.

Figure 5. Cosine Distance Between Test Sentence Vector and Sentence Decode Vectors
To check the similarity, the cosine distance is calculated between Test Sentence Vector (TSV) and each of Sentence Decode Vector (SDV). To calculate similarity, we need to compute the TSV vector, which can be done like this: All the vectors for TSV are taken directly from the vector repository. Now the similarity between TSV and SDV can be calculated by this equation: Where ‖ ‖ denotes Euclidean norm of TSV and ‖ ‖ refers to Euclidean norm of If two vectors are identical than the cosine value will be zero and orthogonal relation between vectors indicates there is no match. The value of angle reaching to zero (or cosine value reaching to one) specifies that the meanings of two words are quite close to each other. The similarity scores are mentioned in the following  The Test Sentence-2 has two words common with Test Sentence-1 i.e. "बडे " and "आम". Similarly, the Test Sentence-3 has "समस्या" and "आम" common with Test Sentence-1. Now, we need to create Context Vectors for Test Sentence 2 and Test Sentence 3: The sense vectors are the same because of the same ambiguous word. However, the Reference Vectors as well as Sentence Decode Vectors will be different for each sentence.

Results
The closest senses of the ambiguous word-"आम" computed by our approach for each of the three abovementioned Test Sentences are shown in Table-  Performance of our proposed approach is compared with a method designed by [17] and shown in Table-6 with setup of same tuning parameters.

Conclusion and future work
This paper proposes a framework to resolve the challenge of ambiguous words using word vectors. The proposed adaptive approach has outperformed the previous approach [17] which is also using word vectors to resolve the ambiguity. The best performance is shown by CBOW model with vector dimensions of 150, window size as 10 and a negative sampling of 5 words. The proposed adaptive approach with the CBOW model and particularly these tuning parameters has shown more than 5% of jump in performance from the previous approach. The lemmatization of words is not performed before training the model which can further enhance the result. Another challenge is to disambiguate words which are used in idioms. Some of these words have completely different interpretations as to their role in idioms which are completely different from their present context.