30201789
Term norm distribution and its effects on Latent Semantic Indexing
Latent Semantic Indexing (LSI) uses the singular value decomposition to reduce noisy dimensions and improve the performance of text retrieval systems. Preliminary results have shown modest improvements in retrieval accuracy and recall, but these have mainly explored small collections. In this paper we investigate text retrieval on a larger document collection (TREC) and focus on distribution of word norm (magnitude). Our results indicate the inadequacy of word representations in LSI space on large collections. We emphasize the query expansion interpretation of LSI and propose an LSI term normalization that achieves better performance on larger collections.
45840953
Dynamic Cluster Formation Using Level Set Methods
Density-based clustering has the advantages for (i) allowing arbitrary shape of cluster and (ii) not requiring the number of clusters as input. However, when clusters touch each other, both the cluster centers and cluster boundaries (as the peaks and valleys of the density distribution) become fuzzy and difficult to determine. In higher dimension, the boundaries become wiggly and over-fitting often occurs. We introduce the notion of cluster intensity function (CIF) which captures the important characteristics of clusters. When clusters are well-separated, CIFs are similar to density functions. But as clusters touch each other, CIFs still clearly reveal cluster centers, cluster boundaries, and, degree of membership of each data point to the cluster that it belongs. Clustering through bump hunting and valley seeking based on these functions are more robust than that based on kernel density functions which are often oscillatory or over-smoothed. These problems of kernel density estimation are resolved using Level Set Methods and related techniques. Comparisons with two existing density-based methods, valley seeking and DBSCAN, are presented to illustrate the advantages of our approach.
5650583
Bipartite graph partitioning and data clustering
Many data types arising from data mining applications can be modeled as bipartite graphs, examples include terms and documents in a text corpus, customers and purchasing items in market basket analysis and reviewers and movies in a movie recommender system. In this paper, we propose a new data clustering method based on partitioning the underlying bipartite graph. The partition is constructed by minimizing a normalized sum of edge weights between unmatched pairs of vertices of the bipartite graph. We show that an approximate solution to the minimization problem can be obtained by computing a partial singular value decomposition (SVD) of the associated edge weight matrix of the bipartite graph. We point out the connection of our clustering algorithm to correspondence analysis used in multivariate analysis. We also briefly discuss the issue of assigning data objects to multiple clusters. In the experimental results, we apply our clustering algorithm to the problem of document clustering to illustrate its effectiveness and efficiency.
1831347
Minimum redundancy feature selection from microarray gene expression data
Selecting a small subset of genes out of the thousands of genes in microarray data is important for accurate classification of phenotypes. Widely used methods typically rank genes according to their differential expressions among phenotypes and pick the top-ranked genes. We observe that feature sets so obtained have certain redundancy and study methods to minimize it. Feature sets obtained through the minimum redundancy - maximum relevance framework represent broader spectrum of characteristics of phenotypes than those obtained through standard ranking methods; they are more robust, generalize well to unseen data, and lead to significantly improved classifications in extensive experiments on 5 gene expressions data sets.
16446006
A similarity-based probability model for latent semantic indexing
A dual probability model is constructed for the Latent Semantic Indexing (LSI) using the cosine similarity measure. Both the document-document similarity matrix and the term-term similarity matrix naturally arise from the maximum likelihood estimation of the model parameters, and the optimal solutions are the latent semantic vectors of of LSI. Dimensionality reduction is justi ed by the statistical signi cance of latent semantic vectors as measured by the likelihood of the model. This leads to a statistical criterion for the optimal semantic dimensions, answering a critical open question in LSI with practical importance. Thus the model establishes a statistical framework for LSI. Ambiguities related to statistical modeling of LSI are clari ed.
4990172
Structure search and stability enhancement of Bayesian networks
Learning Bayesian network structure from large-scale data sets, without any expert-specified ordering of variables, remains a difficult problem. We propose systematic improvements to automatically learn Bayesian network structure from data. (1) We propose a linear parent search method to generate candidate graph. (2) We propose a comprehensive approach to eliminate cycles using minimal likelihood loss, a short cycle first heuristic, and a cut-edge repairing. (3) We propose structure perturbation to assess the stability of the network and a stability-improvement method to refine the network structure. The algorithms are easy to implement and efficient for large networks. Experimental results on two data sets show that our new approach outperforms existing methods.
19735283
A generalized site ranking model for Web IR
Normally, the unit for a ranking model in a Web IR system is a Web page, which is, sometimes, just an information fragment. A larger unit considering the linkage information may be desired to reduce the cognitive overload for users to identify the complete information from the interconnected Web. We propose a ranking model to measure the relevance of the whole Web site. We take some illustrations to show the idea and provide evidences to indicate its effectiveness.
36544447
What Are People Looking for in Your Web Page?
Web server log analyses usually analyze the pattern of the access. We believe that it is also very important to understand the goal of the access. In this paper, we propose to combine the log analysis with content analysis to identify information goals on individual accessed pages. We analyze the web server log to extract information goals on entry pages from anchor texts and query terms, and propagate them along users’ access paths to other linked pages. The experiment shows that our approach could find popular terms on web pages, temporal changes in these terms could reflect users’ interest shifts, and unexpected terms could sometimes indicate a design problem.
31782470
Automatic Keyword Extraction by Server Log Analysis
Traditionally, keywords are extracted from full texts of a document. While in the web environment, there are more sources we can use to provide a more complete view of a web page’s contents. In this paper, we propose to analyze web server logs to extract keywords of entry pages from anchor texts and query terms, and propagate these terms along user access paths to other linked pages. The major benefit of this method is that temporal changes could be reflected in extracted terms, and it is more about a user’s viewpoint on page’s contents instead of author’s.
13344452
Context query in information retrieval
There is an important query requirement missing for search engines. With the wide variation of domain knowledge and user interest, a user would like to retrieve documents in which one query term is discussed in the context of another. Based on existing query mechanisms, what can be specified at most is the co-occurrence of multiple terms in a query. This is insufficient because the co-occurrence of two terms does not necessarily mean that one is discussed in the context of the other. In this paper we propose the context query for Web searching. A new query operator, called the 'in' operator, is used to specify context inclusion between two terms. Heuristic rules to identify context inclusion are suggested and implementation of the 'in' operator in search engines is proposed. Results show that both the precision and ranking relevance of Web searching are improved significantly.
13552447
Reverse mapping of referral links from storage hierarchy for Web documents
Due to the lack of back-pointers, information about the referral parent(s) of a given Web page is not usually available to Web surfers. This significantly reduces the effectiveness of Web surfing and information discovery. We propose a mechanism to locate the referral parent(s) for a given fragment of a Web document (in terms of an URL address) from the client side. The basic idea is to explore the hierarchical hyperlink structure of a Web document fragment from its storage path directory information that is embedded in its URL. Two mechanisms were proposed: exhaustibility first search (EFS) and generalization first search (GFS). Results showed that through direct mapping of the storage hierarchy path to the URL address and then performing reverse tracing along the hyperlink structure, the chance of discovering a referral parent ranged from 41% to 51%. Compared to EFS, GFS was found to reduce the average search space from 26 pages to 4 pages.