Language Independent Sentence-Level Subjectivity Analysis with Feature Selection

Identifying and extracting subjective information from News, Blogs and other user generated content has lot of applications. Most of the earlier work concentrated on English data. But, recently subjectivity related research at sentence-level in other languages has increased. In this paper, we achieve sentence-level subjectivity classification using language independent feature weighing and selection methods which are consistent across languages. Experiments performed on 5 different languages including English and South Asian language Hindi show that Entropy based category coverage difference criterion (ECCD) feature selection method with language independent feature weighing methods outperforms other approaches for subjective classification.


Introduction
Subjective text expresses opinions, emotions, sentiment and beliefs, while objective text generally report facts. So the task of distinguishing subjective from objective text is useful for many natural language processing applications like mining opinions from product reviews (M. Hu and B. Liu, 2004), summarizing different opinions(K. Ganesan.et.al, 2010), question answering (A. Balahur.et.al, 2009) etc.
But research work performed earlier on subjectivity analysis has been applied only on English and mostly at document-level and word-level. Some methods (Wiebe and Riloff, 2005) which concentrated at sentence-level to learn subjective and ob-jective expressions are boot-strapping algorithms which lacks scalability. But, recently focus shifted to multilingual space (R. Mihalcea.et.al, 2007). Banea (C. Banea.et.al, 2008) worked on sentencelevel subjectivity analysis using machine translation approaches by leveraging resources and tools available for English . Another approach (C. Banea.et.al, 2010) used multilingual space and meta classifiers to build high precision classifiers for subjectivity classification.
However, aforementioned work (C. Banea.et.al, 2008) concentrated more on language specific attributes due to variation in expression of subjectivity in different languages. This create a problem of portability of methods to different languages. Other approach (C. Banea.et.al, 2010) which tried achieving language independence created large feature vectors for subjectivity classification. Different languages parallel sentences are taken into consideration to build high-precision classifier for each language. This approach not only increases the complexity and time for classification but also completely dependent on parallel corpus to get good accuracies. A weakly supervised method (C. Lin.et.al, 2011) for sentence-level subjectivity detection using subjLDA tried to reduce training data is available only for English. There are some experiments conducted for Japanese (H. Kanayama.et.al, 2006), Chinese (T. Zagibalov.et.al, 2008), Romanian (C. Banea.et.al, 2008;R. Mihalcea.et.al, 2007) languages data. But these approaches are performed at document level and not language independent.
In this paper, we try to address three major problems highlighted from earlier approaches. First, can language portability problem be eliminated by selecting language independent features. Second, can language specific tools like POS taggers, Named Entity recognizers dependency can be minimized as they vary with language. Third, can accuracy of subjective classification is maintained after feature reduction using feature selection methods which are consistent across languages.
Remainder of this paper is organized into following sections. Related work is mentioned in the Section 2. Next Section 3 discuss about our approach for feature weighing and selection methods. While the experimental setup Section 4 describes collection and evaluation metrics used to analyze the accuracy of approach. Experimental Section 5 explains experiments performed on different languages, while results and performance between SVM and NBM is analyzed in Section 6 and Section 7 respectively. Conclusion and future work is discussed in Section 8.

Related Work
We divide the subjective and objective classification task into unsupervised, multilingual and supervised methods.

Unsupervised
Sentiment classification and opinion analysis can be considered as a hierarchical task of subjectivity detection. Improvement of precision in subjectivity detection can benefit the later. Therefore, lot of work is done for subjective sentence detection to achieve later. (G. Murray.et.al, 2009) proposed to learn subjective expression patterns from both labeled and unlabeled data using n-gram word sequences. Their approach for learning subjective expression patterns is similar to (T. Wilson.et.al, 2008) which relies on n-grams, but goes beyond fixed sequences of words by varying levels of lexical instantiation.

Multilingual
In the multilingual space good amount of work is done in Asian and European languages. Several participants in the Chinese and Japanese Opinion Extraction tasks of NTCIR-6 (Y. Wu.et.al, 2007) performed subjectivity and sentiment analysis in languages other than English. (C. Banea.et.al, 2008;R. Mihalcea.et.al, 2007) performed subjectivity analysis in Romanian. While, (C. Banea.et.al, 2010) performed subjectivity analysis in French, Spanish, Arabic, German, Romanian languages.

Supervised
Furthermore, tools developed for English were used to determine sentiment or subjectivity labeling for a given target language by transferring the text to English and applying an English classifier on the resulting data. The labels were then transfered back into the target language (M. Bautin.et.al, 2008). These experiments are carried out in Arabic, Chinese, English, French, German, Italian, Japanese, Korean, Spanish, and Romanian. (X. Wan, 2009) who constructs a polarity co-training system by using the multi-lingual views obtained through the automatic translation of product-reviews into Chinese and English.

Approach
Subjective sentence classification is treated as a text classification task. (C. Banea.et.al, 2008) used unigrams at word level as features to classify the subjective and objective sentences in different languages. But, unigrams can occur in different categories (subjective and objective) with equal probability. This hampers the classification accuracy. Also, selecting all possible words in the sentences can create a large index size when considered as entire training set increasing the dimensionality of the feature vector for each sentence.
Feature selection can be applied to efficiently categorize the sentences. It is an important process which is followed for many text categorization tasks. Now to achieve our major objective of language independent subjective classification. We use feature extraction, weighing and selection methods that are language independent.

Feature Extraction and Weighing
Features are categorized into syntactic, semantic, link-based, and stylistic features (G. Forman, 2003) from the previous subjective and sentiment studies.
Here, we concentrate more on feature weighing methods based on syntactic and stylistic properties of the text to maintain language independence. Unigrams and Bigrams extracted as features are 172 weighed as given below.

Syntactic Feature Weighing
Syntactic features used in earlier works (M. Gamon, 2004) where word n-grams and part-of-speech (POS) tags. But, POS tagging create dependency on language specific tools. In order to eliminate the language specific dependencies we will use only word n-grams.

Sentence Representation with Unigram (UF.ISF)
This feature extraction is inspired from vector space model (G. Salton, 1975) used for flat documents. UF represents the unigram frequency at word level in a sentence. While ISF represent the inverse sentence frequency of the unigram. For a given collection S of subjective and objective sentences, an Index I = {u 1 , u 2 , ...., u |I| }, where |I| denotes the cardinal of I, gives the list of unigrams u encountered in the sentences S.
A sentence s i of S is then represented by a vector s → i = (w i,1 , w i,2 , ..., w i,I ) followed by the subjective or objective label. Here, w i,j represents the weight of unigram u j in the sentence s i . Now to calculate the weight w i,j we use the formula similar to TF.IDF.
where c i,j is the number of occurrences of u j in the sentence s i normalized by the number of occurrences of other unigrams in sentence s i , |S| is total number of sentences in the training set and |{s i : u j εs i }| is number of sentences in which the unigram u j occurs at-least once.

Sentence Representation with Bigram (BF.ISF)
This feature extraction is similar to UF.ISF mentioned in the earlier section, but we extract co-occurring words. BF represents the Bigrams frequency at word level in a sentence. While ISF represent the inverse sentence frequency of the Bigram. For a given collection S of subjective and objective sentences, an Bigram Index BI = {b 1 , b 2 , ...., b |BI| }, where |BI| denotes the cardinal of BI, gives the list of bigrams b encountered in the sentences S. A sentence s i of S is then represented by a vector s → i = (wb i,1 , wb i,2 , ..., wb i,BI ) followed by the subjective or objective label. Here, wb i,j represents the weight of bigram b j in the sentence s i . Now to calculate the weight wb i,j we use the formula similar to UF.ISF.
where c i,j is the number of occurrences of b j in the sentence s i normalized by the number of occurrences of other bigrams in sentence s i , |S| is total number of sentences in the training set and |{s i : b j εs i }| is number of sentences in which the bigram b j occurs at least once.

Stylistic Feature Weighing
Structural and lexical style markers can be considered as stylistic features which has shown good results in Web discourse (A. Abbasi.et.al, 2008). However, style markers have seen limited usage in sentiment analysis research. Some (M. Gamon, 2004) tried in this direction.

Sentence representation with Normalized Unigram Word Length (NUWL)
This feature extraction considers length of unique unigram words in the sentence. Length of unigram is calculated by the number of characters present in the word. For a given collection S of subjective and objective sentences, an Word Index W I = {uw 1 , uw 2 , ...., uw |W I| }, where |W I| denotes the cardinal of W I, gives the list of unigram words uw encountered in the sentences S. A sentence s i of S is then represented by a vector s → i = (lw i,1 , lw i,2 , ..., lw i,I ) followed by the subjective or objective label. Here, lw i,j represents the weight of unigram word uw j in the sentence s i . Now to calculate the weight lw i,j .
where L i,j is the character count in the uw j in the sentence s i normalized by length of all the unigram words in sentence s i . |S| is total number of sentences in the training set and |{s i : uw j εs i }| is number of sentences in which the unigram uw j occurs atleast once.

Feature Selection
Feature selection methods help in removing the features which may not be useful for categorization. To achieve it, feature selection techniques select subset of total features. But, it is important to reduce features without compromising on the accuracy of a classifier. Most methods like Information Gain(IG) (C. Lee.et.al, 2006), Correlation Feature Selection(CFS) (M.A. Hall, 1999), Chi-Squared (χ 2 ) (J. Bakus.et.al, 2006), Odds ratio (OR) (G. Forman, 2003) does not consider the frequency of the text or term between the categories which leads in reduction of accuracy of a classifier.
In-order to overcome this problem, we used Entropy based category coverage difference (ECCD) (C. Largeron.et.al, 2011) feature selection method which uses the entropy of the text or term. f j is used to represent the text feature extracted (unigram or bigram), c k for category of the class and c − k for the complement of the class. Where j represent number of features and k represents two classes either subjective or objective.
Entropy based category coverage Difference(ECCD) This feature selection method (C. Largeron.et.al, 2011) was proposed to mine INEX XML documents. We use this approach for improving the subjective and objective sentence classification. Let T k j be number of occurrences of text feature f j in the category c k sentence and, f q k j is the frequency of f j in that category c k given by Equation 4.
So Entropy Ent(f j ) of text feature f j is given by Equation 5 Ent Entropy equals 0, if the text feature f j appears only in one category. It means that feature has good discrimination ability to classify the sentences. Similarly, entropy of the text feature will be high if the feature is represented in two classes. If Ent m represent the maximum entropy of the feature f j , ECCD(f j , c k ) is given by following equation 6.
Where P (f j |c k ) and P (f j |c − k ) are probability of observing the text feature f j in a sentence belonging to category c k and c − k respectively. The advantage of ECCD is that higher the number of sentences of category c k containing feature f j and lower the number of sentences in other category containing f j , we get higher value for equation 6. It means f j becomes the characteristic feature of that category c k which helps in better feature selection. Feature selection method which is similar to ECCD is mentioned below.
Categorical Proportional Difference (CPD) CPD (M. Simeon.et.al, 2008) is a measure of the degree to which a text feature contributes to differentiating a particular category from other categories in a text corpus. We calculate the CPD for a text feature f j by taking a ratio that considers the number of sentences in subjective category c k in which the text feature occurs and the number of sentences in objective category c − k in which the text f j also occurs. Equation 7 shows the details. Certain threshold of CPD score is kept to reduce the number of features.

Contingency Table Representation of features
Feature selection methods mentioned in earlier section is estimated using a contingency table. Let A, be the number of sentences in the subjective category containing feature f j . B, be the number of sentences in the objective category containing f j . C, be the number of sentences of subjective category which do not contain feature f j given by f − j and D, be the number of sentences in objective category c − k which do not contain f j . Let (M = A + B + C + D) be the total possibilities.  Table   FS Representation  Table   4 Experimental Setup In-order to achieve subjective and objective classification at sentence level for different languages. We performed our experiments using different datasets.

Datasets
Translated MPQA corpus provided in (C. Banea.et.al, 2010) containing subjective and objective sentences 1 of French, Arabic, and Romanian languages were used for experiments. For English, MPQA corpus 2 containing subjective and objective sentences used for translation of above mentioned corpus is used. Hindi experiments were performed using sentences from the news corpus (A. Mogadala.et.al, 2012) tagged with positive, negative and objective sentences. Positive and negative sentences are further clubbed into subjective sentences to do subjectivity analysis.

Evaluation
To evaluate various feature selection methods, we use F-measure scores which combines precision and recall. Precision (P s ) measures the percentage of sentences correctly assigned to subjective category, and recall (R s ) measures the percentage of sentences that should have been assigned to subjective category but actually assigned to subjective category. Using P s and R s subjective F-measure F s is calculated. Similarly, Objective F-measure F o is cal-culated using P o and R o . After F-measure is determined for both subjective and objective class, the macro-average F-measure F macro−avg is determined by the following Equation 8.

Experiments
Initially, 1500 subjective and 1500 objective sentences of English, Romanian, French and Arabic languages are used to perform the experiments. While for Hindi, entire corpus constituting 786 subjective and 519 objective sentences was used. Different feature weighing and selection methods are evaluated with 2 different classifiers to obtain best combination for each language. Table 8 to Table 12 show the Macro-Average (F macro−avg ) scores obtained after 10 cross-validation using sequential minimal optimization algorithm (J. Platt, 1998) for training a support vector machine(SVM) using polynomial kernel and Naive Bayes Multinomial(NBM) classifiers. Feature space obtained after application of feature selection methods for each language are mentioned in Tables 3, 4, 5, 6, 7. Once the best combination is obtained for each language. It is compared with multilingual space classifier proposed in (C. Banea.et.al, 2010) 3 along with the baseline constituting simple Naive Bayes classifier with unigram features. Multilingual space constitutes words as features from all languages used for experiments except Hindi.
Scalability of feature selection methods is an issue. In-order to understand the performance of ECCD feature selection method with classifiers. In every iteration 500 sentences are added to each class of initial 1500 subjective and objective sentences limiting to maximum of 3500 to get average scores. Table 13 show the comparison of average scores obtained for each language. Figures 1, 2, 3, 4 show precision and recall for subjective sentences obtained using different methods for English, Romanian, French and Arabic respectively using different number of sentences.

Result Analysis
It is observed from the Table 8 to Table 12 that ECCD feature selection and BF.ISF feature weigh-ing method with NBM classifier performs consistently across languages. This behavior is observed due to capability of ECCD in efficiently discrimi-       nating the features belonging to a particular class. Although, UF.ISF and NUWL with ECCD and CPD using SVM classifier has more scores than BF.ISF, it is not significantly contrasting with the results of NBM classifier. So, NBM classifier with ECCD feature selection and BF.ISF feature weighing method which obtained high F macro−avg scores is selected for comparison with other approaches in Table 13. The proposed method not only outperforms on F macro−avg compared to other approaches but also on P macro−avg and R macro−avg in all languages. For English, proposed methods gains 23.8% over baseline in F macro−avg and 8.0% on P macro−avg on (Wiebe and Riloff, 2005). Similarly, from Table 13 it can be deducted that proposed method for Romanian attains 26.1% more F macro−avg than baseline and 155.4% more compared to Multilingual space classifier (C. Banea.et.al, 2010). Similar observations can be made for other languages. Even though (C. Banea.et.al, 2010) attains high recall for every language. It fails to attain high precision due to presence of large number of frivolous word features which are common for both classes. This being major drawback, ECCD feature selection method eliminates features which attains zero entropy. This reduces the randomness of features and leave only those features which are more eligible for discriminating the classes. Combining ECCD with BF.ISF, a language independent weighing method for Bi-gram features extracted from the sentences. We are able to attain a best classification accuracies which are consistent across languages. Figures 1, 2, 3, 4 also show that increase in number of sentences does not effect the precision of the proposed method, as it still outperforms other methods. But, scalability problem persists for ECCD with BF.ISF for larger datasets, as it may not eliminate less random features due to noise and other constraints. Also, feature selection methods ensure the performance of classifiers is maintained by reducing number of features. But, it does not ensure reduction in fixed percentage of features. As observed our best performing feature selection method ECCD reduces feature size by 10% only.

Performance Analysis between SVM and NBM
From the Table 8 to 12 it is observed that SVM with some feature selection and weighing methods performs equivalent to the NBM. However, as the number of documents increases the performance of SVM may degrade. It can be derived that, as the training data size increases, it is rare to see SVM performing better than NBM.

Training Time behavior
SVM is in a clear disadvantage compared to NBM when processing time is considered. The training time of the SVM is particularly high, especially for larger feature spaces. It is probably attributed to the time taken in finding the proper separating hyperplane.

Features behavior
Large feature spaces do not necessarily lead to best performance. So feature selection methods are used to create small feature spaces to build SVM and NBM classifiers. Sometimes, small feature space sizes make SVM perform equivalent to NBM as observed in Table 12. Thus, this would explain why SVM is outperformed for small training set sizes and for small feature spaces with large training sets.

Conclusion and Future Work
In this paper, subjective classification is achieved using combination of feature selection and weighing methods which are consistent across languages. We found that our proposed method which combines ECCD feature selection and BF.ISF feature weighing method used along with NBM classifier perform across languages. It not only outperforms other feature selection methods but also achieve better scores compared to other approaches. In future, we want to apply this approach on bigger datasets and also extend it to multiple class problems.