Low-Quality Product Review Detection in Opinion Summarization

Product reviews posted at online shopping sites vary greatly in quality. This paper addresses the problem of detecting lowquality product reviews. Three types of biases in the existing evaluation standard of product reviews are discovered. To assess the quality of product reviews, a set of specifications for judging the quality of reviews is first defined. A classificationbased approach is proposed to detect the low-quality reviews. We apply the proposed approach to enhance opinion summarization in a two-stage framework. Experimental results show that the proposed approach effectively (1) discriminates lowquality reviews from high-quality ones and (2) enhances the task of opinion summarization by detecting and filtering lowquality reviews.


Introduction
In the past few years, there has been an increasing interest in mining opinions from product reviews (Pang, et al, 2002;Liu, et al, 2004;Popescu and Etzioni, 2005). However, due to the lack of editorial and quality control, reviews on products vary greatly in quality. Thus, it is crucial to have a mechanism capable of assessing the quality of reviews and detecting low-quality/noisy reviews.
Some shopping sites already provide a function of assessing the quality of reviews. For example, Amazon 1 allows users to vote for the helpfulness of each review and then ranks the reviews based on the accumulated votes. However, according to our survey in Section 3, users' votes at Amazon have three kinds of biases as follows: (1) imbalance vote bias, (2) winner circle bias, and (3) early bird bias. Existing studies (Kim et al, 2006;Zhang and Varadarajan, 2006) used these users' votes for training ranking models to assess the quality of reviews, which therefore are subject to these biases.
In this paper, we demonstrate the aforementioned biases and define a standard specification to measure the quality of product reviews. We then manually annotate a set of ground-truth with real world product review data conforming to the specification.
To automatically detect low-quality product reviews, we propose a classification-based approach learned from the annotated ground-truth. The proposed approach explores three aspects of product reviews, namely informativeness, readability, and subjectiveness.
We apply the proposed approach to opinion summarization, a typical opinion mining task. The proposed approach enhances the existing work in a two-stage framework, where the low-quality review detection is applied right before the summarization stage.
Experimental results show that the proposed approach can discriminate low-quality reviews from high-quality ones effectively. In addition, the task of opinion summarization can be enhanced by detecting and filtering low-quality reviews.
The rest of the paper is organized as follows: Section 2 introduces the related work. In Section 3, we define the quality of product reviews. In Section 4, we present our approach to detecting lowquality reviews. In Section 5, we empirically verify the effectiveness of the proposed approach and its use for opinion summarization. Section 6 summarizes our work in this paper and points out the future work.

Evaluating Helpfulness of Reviews
The problem of evaluating helpfulness of reviews (Kim et al, 2006), also known as learning utility of reviews (Zhang and Varadarajan, 2006), is quite similar to our problem of assessing the quality of reviews.
In practice, researchers in this area considered the problem as a ranking problem and solved it with regression models. In the process of model training and testing, they used the ground-truth derived from users' votes of helpfulness provided by Amazon. As we will show later in Section 3, these models all suffered from three types of voting bias.
In our work, we avoid using users' votes by developing a specification on the quality of reviews and building a ground-truth according to the specification.

Mining Opinions from Reviews
One area of research on opinion mining from product reviews is to judge whether a review expresses a positive or a negative opinion. For example, Turney (2006) presented a simple unsupervised learning algorithm in judging reviews as "thumbs up" (recommended) or "thumbs down" (not recommended). Pang et al (2002) considered the same problem and presented a set of supervised machine learning approaches to it. For other work see also Dave et al. (2003), Lee (2004, 2005).
Another area of research on opinion mining is to extract and summarize users' opinions from product reviews (Hu and Liu, 2004;Liu et al., 2005;Popescu and Etzioni, 2005). Typically, a sentence or a text segment in the reviews is treated as the basic unit. The polarity of users' sentiments on a product feature in each unit is extracted. Then the aggregation of the polarities of individual senti-ments is presented to users so that they can have an at-a-glance view on how other experienced users rated on a certain product. The major weakness in the existing studies is that all the reviews, including low-quality ones, are taken into consideration and treated equally for generating the summary. In this paper, we enhance the application by detecting and filtering low-quality reviews. In order to achieve that, we first define what the quality of reviews is.

Quality of Product Reviews
In this section, we will first show three biases of users' votes observed on Amazon, and then present our specification on the quality of product reviews.

Amazon Ground-truth
In our study, we use the product reviews on digital cameras crawled from Amazon as our data set. The data set consists of 23,141 reviews on 946 digital cameras. At the Amazon site, users could vote for a review with a "helpful" or "unhelpful" label. Thus, for each review there are two numbers indicating the statistics of these two labels, namely the number of "helpful" votes and that of "unhelpful" ones. Kim et al (2006) used the percentage of "helpful" votes as the measure of evaluating the "quality of reviews" in their experiments. We call the ground-truth based on this measure as "Amazon ground-truth".
Certainly, the ground-truth has the advantage of convenience. However, we identify three types of biases that make the Amazon ground-truth not always suitable for determining the quality of reviews. We describe these biases in details in the rest of this section. reviews (corresponding to the two bars on the right of the figure) have more than 90% "helpful" votes, including 9,100 reviews with 100% "helpful" votes. From an in-depth investigation on these highly-voted reviews, we observed that some did not really have as good quality as the votes hint. For example, in Figure 2, the review about Canon PowerShot S500 receives 40 "helpful" votes out of 40 votes although it only gives very brief description on the product features in its second paragraph. We call this type of bias "imbalance vote" bias.   There also exists a bootstrapping effect of "hot" reviews at the Amazon site. Figure 3 shows the "helpful" votes for the top 50 ranked reviews. The numbers are averaged over 127 digital cameras which have no less than 50 reviews. As shown in this figure, the top two reviews hold more than 250 and 140 votes respectively on average; while the numbers of votes held by lower-ranked reviews decrease exponentially. This is so-called the "winner circle" bias: the more votes a review gains, the more default authority it would appear to the readers, which in turn will influence the objectivity of the readers' votes. Also, the higher ranked reviews would attract more eyeballs and therefore gain more people's votes. This mutual influence among labelers should be avoided when the votes are used as the evaluation standard.

Figure 4. Dependency on publication date
Publication date can influence the accumulation of users' votes. In Figure 4, the n'th publication date represents the n'th month after the product is released. The number in the figure is averaged over all the digital cameras in the data set. We can observe a clear trend that the earlier a review is posted, the more votes it will get. This is simply because reviews posted earlier are exposed to users for a longer time. Therefore, some high quality reviews may get fewer users' vote because of later publication. We call this "early bird" bias.

Specification of Quality
Besides these aforementioned biases, using the raw rating from readers directly also fails to provide a clear guideline for what a good review consists of. In this section, we provide such a guideline, which we name as the specification (SPEC).
In the SPEC, we define four categories of review quality which represent different values of the reviews to users' purchase decision: "best review", "good review", "fair review", and "bad review". A generic description of the SPEC is as follows: A best review must be a rather complete and detailed comment on a product. It presents several aspects of a product and provides convincing opinions with enough evidence. Usually a best review could be taken as the main reference that users only need to read before making their purchase decision on a certain product. The first review in Figure 5 is a best review. It presents several product features and provides convincing opinions with sufficient evidence. It is also in a good format for readers to easily understand. Note that we omit some words in the example to save the space. A good review is a relatively complete comment on a product, but not with as much supporting evidence as necessary. It could be used as a strong and influential reference, but not as the only recommendation. The second review in Figure 5 is such an example.
A fair review contains a very brief description on a product. It does not supply detailed evaluation on the product, but only comments on some aspects of the product. For example, the third review in Figure 5 mainly talks about "the delay between pictures", but less about other aspects of the camera.
A bad review is usually an incorrect description of a product with misleading information. It talks little about a specific product but much about some general topics (e.g. photography). For example, the last review in Figure 5 talks about the topic of "generic battery", but does not specify any digital camera. A bad review is an "unhelpful" review that can be ignored. There is nothing wrong with the 2100 except for the very noticeable delay between pics. The camera's digital processor takes about 5 seconds after a photo is snapped to ready itself for the next one. Otherwise, the optics, the 3X optical zoom and the 2 megapixel resolution are fine for anything from Internet apps to 8" x 10" print enlarging. It is competent, not spectacular, but it gets the job done at an agreeable price point.

Annotation of Quality
According to the SPEC defined above, we built a ground-truth from the Amazon data set. We randomly selected 100 digital cameras and 50 reviews for each camera. Totally we have 4,909 reviews since some digital cameras have fewer than 50 unique reviews. Then we hired two annotators to label the reviews with the SPEC as their guideline. As the result, we have two independent copies of annotations on 4,909 reviews, with the labels of "best", "good", "fair", and "bad". Table 1 shows the confusion matrix between the two copies of annotation. The value of the kappa statistic (Cohen, 1960) calculated from the matrix is 0.8142. This shows that the two annotators achieved highly consistent results by following the SPEC, although they worked independently. In order to examine the difference between our annotations and Amazon ground-truth, we evaluate the Amazon ground-truth against the annotations, with the measure of "error rate of preference pairs" (Herbrich et al, 1999).
where the "preference pair" is defined as a pair of reviews with a order. For example, a best review and a good review correspond to a preference pair with the order of "best review preferring to good review". The "all preference pairs" are collected from one of the annotations (the annotation 1 or the annotation 2) by ignoring the pairs from the same category. The "incorrect preference pairs" are the preference pairs collected from the Amazon ground-truth but not with the same order as that in the all preference pairs. The order of the preference pair collected from the Amazon ground-truth is evaluated on the basis of the percentage score as described in Section 3.1. The error rate of preference pairs based on the annotation 1 and that based on the annotation 2 are 0.448 and 0.446, respectively, averaged over 100 digital cameras. The high error rate of preference pairs demonstrates that the Amazon ground-truth diverges from the annotations (our ground-truth) significantly.
To discover which kind of ground-truth is more reasonable, we ask an additional annotator (the third annotator) to compare these two kinds of ground-truth. More specifically, we randomly selected 100 preference pairs whose orders the two kinds of ground-truth don't agree on (called incorrect preference pairs in the evaluation above). As for our ground-truth, we choose the Annotation 1 in the new test. Then, the third annotator is asked to assign a preference order for each selected pair. Note that the third annotator is blind to both our specification and the existing preference order. Last, we evaluate the two kinds of ground-truth with the new annotation. Among 100 pairs, our ground-truth agrees to the new annotation on 85 pairs while the Amazon ground-truth agrees to the new annotation on 15 pairs. To confirm the result, yet another annotator (the fourth annotator) is called to repeat the same annotation independently as the third one. And we obtain the same statistical result (85 vs. 15) although the fourth annotator does not agree with the third annotator on some pairs.
In practice, we treat the reviews in the first three categories ("best", "good" and "fair") as highquality reviews and those in the "bad" category as low-quality reviews, since our goal is to identify low quality reviews that should not be considered when creating product review summaries.

Classification of Product Reviews
We employ a statistical machine learning approach to address the problem of detecting low-quality products reviews.
Given a training data set = , 1 , we construct a model that can minimize the error in prediction of y given x (generalization error). Here ∈ and = { , } represents a product review and a label, respectively. When applied to a new instance x, the model predicts the corresponding y and outputs the score of the prediction.

The Learning Model
In our study, we focus on differentiating lowquality product reviews from high-quality ones. Thus, we treat the task as a binary classification problem.
We employ SVM (Support Vector Machines) (Vapnik, 1995) as the model of classification. Given an instance x (product review), SVM assigns a score to it based on = + (2) where w denotes a vector of weights and b denotes an intercept. The higher the value of f(x) is, the higher the quality of the instance x is. In classification, the sign of f(x) is used. If it is positive, then x is classified into the positive category (high-quality reviews), otherwise into the negative category (low-quality reviews).
The construction of SVM needs labeled training data (in our case, the categories are "high-quality reviews" and "low-quality reviews"). Briefly, the learning algorithm creates the "hyper plane" in (2), such that the hyper plane separates the positive and negative instances in the training data with the largest "margin".

Product Feature Resolution
Product features (e.g., "image quality" for digital camera) in a review are good indicators of review quality. However, different product features may refer to the same meaning (e.g., "battery life" and "power"), which will bring redundancy in the study. In this paper, we formulize the problem as the "resolution of product features". Thus, the problem is reduced to how to determine the equivalence of a product feature in different forms.
In (Hu and Liu, 2004), the matching of different product features is mentioned briefly and addressed by fuzzy matching. However, there exist many cases where the method fails to match the multiple mentions, e.g., "battery life" and "power", because it only considers string similarity. In this paper we propose to resolve the problem by leveraging two kinds of evidence: one is "surface string" evidence, the other is "contextual evidence".
We use edit distance (Ukkonen, 1985) to compare the similarity between the surface strings of two mentions, and use contextual similarity to reflect the semantic similarity between two mentions.
When using contextual similarity, we split all the reviews into sentences. For each mention of a product feature, we take it as a query and search for all the relevant sentences. Then we construct a vector for the mention, by taking each unique term in the relevant sentences as a dimension of the vector. The cosine similarity between two vectors of mentions is then present to measure the contextual similarity between two mentions.

Feature Development for Learning
To detect low-quality reviews, our proposed approach explores three aspects of product reviews, namely informativeness, subjectiveness, and readability. We denote the features employed for learning as "learning features", discriminative from the "product features" we discussed above.

Features on Informativeness
As for informativeness, the resolution of product features is employed when we generate the learning features as listed below. Pairs mapping to the same product feature will be treated as the same product feature, when we calculate the frequency and the number of product features. We apply the approach proposed in (Hu and Liu, 2004) to extract product features.
We also use a list of product names and a list of brand names to generate the learning features. Both lists can be collected from the Amazon site because they are relatively stable within a time interval.
The learning features on the informativeness of a review are as follows.  Sentence level (SL)  The number of sentences in the review  The average length of sentences  The number of sentences with product features  Word level (WL)  The number of words in the review  The number of products (e.g., DMC-FZ50, EX-Z1000) in the review  The number of products in the title of a review  The number of brand names (e.g., Canon, Sony) in the review  The number of brand names in the title of a review  Product feature level (PFL)  The number of product features in the review  The total frequency of product features in the review  The average frequency of product features in the review  The number of product features in the title of a review  The total frequency of product features in the title of a review

Features on Readability
We make use of several features at paragraph level which indicate the underlying structure of the reviews. These features include,  The number of paragraphs in the review  The average length of paragraphs in the review  The number of paragraph separators in the review Here, we refer to the keywords, such as "Pros" vs. "Cons" as "paragraph separators". The keywords usually appear at the beginning of paragraphs for categorizing two contrasting aspects of a product. We extract the nouns and noun phrases at the beginning of each paragraph from the 4,909 reviews and use the most frequent 30 pairs of keywords as paragraph separators. Table 2 provides some examples of the extracted separators.

Features on Subjectiveness
We also take the subjectiveness of reviews into consideration. Unlike previous work (Kim et al, 2006;Zhang and Varadarajan, 2006) using shallow syntactic information directly, we use a sentiment analysis tool (Hu and Liu, 2004) which aggregates a set of shallow syntactic information. The tool is a classifier capable of determining the sentiment polarity of each sentence. We create three learning features regarding the subjectiveness of reviews.  The percentage of positive sentences in the review  The percentage of negative sentences in the review  The percentage of subjective sentences (regardless of positive or negative) in the review

Experiments
In this section, we describe our experiments with the proposed classification-based approach to lowquality review detection, and its effectiveness on the task of opinion summarization.

Detecting Low-quality Reviews
In our proposed approach, the problem of assessing quality of reviews is formalized as a binary classification problem. We conduct experiments by taking reviews in the categories of "best", "good", and "fair" as high-quality reviews and those in the "bad" category as low-quality reviews. As for classification model, we utilize the SVMLight toolkit (Joachims, 2004). We randomly divide the 100 queries of digital cameras into two sets, namely a training set of 50 queries and a test set of 50 queries. For the two copies of annotations, we use the same division. We use the training set from "annotation 1" to train the model and apply the model to the test sets from both "annotation 1" and "annotation 2", respectively. Table 3 reports the accuracies of our approach to review classification. The accuracy is defined as the percentage of correctly classified reviews.
We take the approach that utilizes only the category of features on sentence level (SL) as the baseline, and incrementally add other categories of features on informativeness, readability and subjectiveness. We can see that both the features on word level (WL) and those on product feature level (PFL) can improve the performance of classification much. The features on readability can still increase the accuracy although the contribution is much less. The features on subjectiveness, however, make no contribution.  Table 3. Low-quality reviews detection We also conduct a more detailed analysis on each individual feature. Two categories of features on "title" and "brand name" have poor performance, which is due to the lack of information in the title and the low coverage of brand names in a review, respectively.

Summarizing Sentiments of Reviews
One potential application of low-quality review detection is the opinion summarization of reviews.
The process of opinion summarization of reviews with regards to a query of a product consists of the following steps (Liu et al, 2005): 1. From each of the reviews, identify every text segment with opinion in the review, and determine the polarities of the opinion segments.
2. For each product feature, generate a positive opinion set and a negative opinion set of opinion segments, denoted as POS( ) and NOS( ).
3. For each product feature, aggregate the numbers of segments in POS( ) andNOS( ), as opinion summarization on the product feature.
In this process, all the reviews contribute the same. However, different reviews do hold different authorities. A positive/negative opinion from a high-quality review should not have the same weight as that from a low-quality review.
We use a two-stage approach to enhance the reliability of summarization. That is, we add a process of low-quality review detection before the summarization process, so that the summarization result is obtained based on the high-quality reviews only. We are to demonstrate how much difference the proposed two-stage approach can bring into the opinion summarization.
We use the best classification model trained as described in Section 5.1 to filter low-quality reviews, and do summarization on the high-quality reviews associated to the 50 test queries. We denote the proposed approach and the old approach as "two-stage" and "one-stage", respectively. Due to the limited space, we only give a visual comparison of the two approaches on "image quality" in Figure 6. The upper figure shows the summarization of positive opinions and the lower figure shows that of negative opinions. From the figures we can see that the two-stage approach preserves fewer text segments as the result of filtering out many low-quality product reviews.

Figure 6. Summarization on "image quality"
To show the comparison on more features in a compressed space, we give the statistic ratio of change between two approaches instead. As for the evaluation measure, we define "RatioOfChange" (ROC) on a feature f as, where Rate * (f) is defined as, (4) Table 4 shows some statistic results on ROC on five product features, namely "image quality"(IQ), "battery", "LCD screen" (LCD), "flash" and "movie mode" (MM). The values in the cells are the percentage of queries whose ROC is larger/smaller than the respective thresholds. We can see that a large portion of queries have big changes on the values of ROC. This means that the result achieved by the two-stage approach is substantially different from that achieved by the one-stage approach. There is no standard way to evaluate the quality of opinion summarization as it is rather a subjective problem. In order to demonstrate the impact of the two-stage approach, we turn to external authoritative sources other than Amazon.com as the objective evaluation reference. We observe that CNET 2 provides a professional "editor's review" for many products, which gives a rating in the range of 1~10 on product features. 9 digital cameras out of the 50 test queries are found to have the editor's rating on "image quality" at CNET. We use this rating to compare with the results of our opinion summarization. We rescale the Rate scores obtained by both the one-stage approach and the two-stage approach into the range of 1-10 in order to perform the comparison. Figure 7 provides the visual comparison. We can see that the result achieved by the two-stage approach has a much better (closer) resemblance to CNET rating than one-stage approach does. This indicates that our two-stage approach can achieve a more consistent summarization result to the professional evaluations by the editors. Although the CNET rating is not the absolute standard for product evaluation, it provides a professional yet objective evaluation of the products. Therefore, the experimental results demonstrate that our proposed approach could achieve more reliable opinion summarization which is closer to the generic evaluation from authoritative sources. In this paper, we studied the problem of detecting low-quality product reviews. Our contribution can be summarized in two-fold: (1) we discovered three types of biases in the ground-truth used extensively in the existing work, and proposed a specification on the quality of product reviews. The three biases that we discovered are imbalance vote bias, winner circle bias, and early bird bias.
(2) Rooting on the new ground-truth (conforming to the proposed specification), we proposed a classification-based approach to low-quality product review detection, which yields better performance of opinion summarization. We hope to explore our future work in several areas, such as further consolidating the new ground-truth from different points of view and verifying the effectiveness of low-quality review detection with other applications.