1490908
Word segmentation and recognition for web document framework
It is observed that a better approach to Web information understanding is to base on its document framework, which is mainly consisted of (i) the title and the URL name of the page, (ii) the titles and the URL names of the Web pages that it points to, (iii) the alternative information source for the embedded Web objects, and (iv) its linkage to other Web pages of the same document. Investigation reveals that a high percentage of words inside the document framework are “compound words” which cannot be understood by ordinary dictionaries. They might be abbreviations or acronyms, or concatenations of several (partial) words. To recover the content hierarchy of Web documents, we propose a new word segmentation and recognition mechanism to understand the information derived from the Web document framework. A maximal bi-directional matching algorithm with heuristic rules is used to resolve ambiguous segmentation and meaning in compound words. An adaptive training process is further employed to build a dictionary of recognisable abbreviations and acronyms. Empirical results show that over 75% of the compound words found in the Web document framework can be understood by our mechanism. With the training process, the success rate of recognising compound words can be increased to about 90%.
9546816
A Framework of HTML Content Selection and Adaptive Delivery
In this paper, we propose a content selection and adaptive delivery framework to adapt content delivery to different client capabilities and preferences. The framework provides language support to content publishers with XML-based markup language to describe the content and control the transformation and delivery process. Automatic content classification and selective delivery based on match between content class and user capability or preference are also implemented. Thus, when content descriptions from server are not available, adaptive content delivery is still possible. The framework is generic and scalable. Performance studies on content classification and selection model show that it is effective in saving network resources and improving clients’ access experiences.
10076076
Quantitative Modeling for Web Objects' Cacheability
In this paper, we propose a mathematical model, called the E-Cacheability Index, to quantify and compare the effectiveness of caching web objects on Internet. Unlike previous related work on web object cacheability, our study is based on a comprehensive study of all object attributes related to the HTTP protocol transfer, content reuse, freshness re-validation as well as the inter-relationship/dependency among these attributes in the final caching decision. This model is important because it not only estimates the cacheability effectiveness of web content, but also provides hints on web caching optimization.
13552447
Reverse mapping of referral links from storage hierarchy for Web documents
Due to the lack of back-pointers, information about the referral parent(s) of a given Web page is not usually available to Web surfers. This significantly reduces the effectiveness of Web surfing and information discovery. We propose a mechanism to locate the referral parent(s) for a given fragment of a Web document (in terms of an URL address) from the client side. The basic idea is to explore the hierarchical hyperlink structure of a Web document fragment from its storage path directory information that is embedded in its URL. Two mechanisms were proposed: exhaustibility first search (EFS) and generalization first search (GFS). Results showed that through direct mapping of the storage hierarchy path to the URL address and then performing reverse tracing along the hyperlink structure, the chance of discovering a referral parent ranged from 41% to 51%. Compared to EFS, GFS was found to reduce the average search space from 26 pages to 4 pages.
18215853
Study for fusion of different sources to determine relevance
The relevance of a Web document could be measured not only by its text content, but also by some other factors such as the link connectivity and usage patterns. In previous data fusion researches, the text is the only source to determine the relevance, and only the different runs (e.g. by different retrieval models, different query or document representations) on this same source are combined. It is the purpose of this paper to investigate whether the different sources can be combined to determine the relevance with a better accuracy than any single source. We conducted a preliminary experiment to test its feasibility and effectiveness and a positive result was obtained.
209398610
Beyond the traditional query operators.
13840611
Dynamic cluster formation using level set methods
Density-based clustering has the advantages for: 1) allowing arbitrary shape of cluster and 2) not requiring the number of clusters as input. However, when clusters touch each other, both the cluster centers and cluster boundaries (as the peaks and valleys of the density distribution) become fuzzy and difficult to determine. We introduce the notion of cluster intensity function (CIF) which captures the important characteristics of clusters. When clusters are well-separated, CIFs are similar to density functions. But, when clusters become closed to each other, CIFs still clearly reveal cluster centers, cluster boundaries, and degree of membership of each data point to the cluster that it belongs. Clustering through bump hunting and valley seeking based on these functions are more robust than that based on density functions obtained by kernel density estimation, which are often oscillatory or oversmoothed. These problems of kernel density estimation are resolved using level set methods and related techniques. Comparisons with two existing density-based methods, valley seeking and DBSCAN, are presented which illustrate the advantages of our approach.
11783493
Unsupervised Feature Selection Via Two-way Ordering in Gene Expression Analysis
MOTIVATION
Selection of genes most relevant and informative for certain phenotypes is an important aspect in gene expression analysis. Most current methods select genes based on known phenotype information. However, certain set of genes may correspond to new phenotypes which are yet unknown, and it is important to develop novel effective selection methods for their discovery without using any prior phenotype information.


RESULTS
We propose and study a new method to select relevant genes based on their similarity information only. The method relies on a mechanism for discarding irrelevant genes. A two-way ordering of gene expression data can force irrelevant genes towards the middle in the ordering and thus can be discarded. Mechanisms based on variance and principal component analysis are also studied. When applied to expression profiles of colon cancer and leukemia, the unsupervised method outperforms the baseline algorithm that simply uses all genes, and it also selects relevant genes close to those selected using supervised methods.


SUPPLEMENT
More results and software are online: http://www.nersc.gov/~cding/2way.
270633
Web document clustering using hyperlink structures
WEB DOCUMENT CLUSTERING USING HYPERLINK STRUCTURES XIAOFENG HE y , HONGYUAN ZHA , CHRIS H.Q. DING y AND HORST D. SIMON y Abstract. With the exponential growth of information on the World Wide Web, there is great demand for developing e cient and e ective methods for organizing and retrieving the information available. Document clustering plays an important role in information retrieval and taxonomy man- agement for the World Wide Web and remains an interesting and challenging problem in the eld of web computing. In this paper we consider document clustering methods exploring textual infor- mation, hyperlink structure and co-citation relations. In particular, we apply the normalized-cut clustering method developed in computer vision to the task of hyperdocument clustering. We also explore some theoretical connections of the normalized-cut method to K-means method. We then experiment with normalized-cut method in the context of clustering query result sets for web search engines. Keywords. World Wide Web, graph partitioning, cheeger constant, clustering method, K-means method, normalized cut method, eigenvalue decomposition, power method. 1. Introduction. Currently the World Wide Web contains billions of documents and it is still growing rapidly. Finding the relevant documents to satisfy a user's infor- mation need is a very important and challenging task. Many commercial search en- gines have been developed and used by millions of people all over the world. However, the relevancy of documents returned in search engine result sets is still lacking, and further research and development is needed to really make search engines a ubiquitous information-seeking tool. The World Wide Web has a rich structure: it contains both textual web documents and the hyperlinks that connect them. The web documents and hyperlinks between them form a directed graph in which the web documents can be viewed as vertices and the hyperlinks as directed edges. Algorithms have been de- veloped utilizing this directed graph to extract information contained in a collection of hyperlinked web documents. Kleinberg proposed HITS algorithm based purely on hyperlink information to retrieve the most relevant information: authority and hub documents for a user query 20]. However, if the hypertext collection consists of sev- eral topics, authority and hub documents may only cover the most popular topics and leave out the less popular ones. One way to remedy this situation is to rst partition the hypertext collection into topical groups, and present the search results as a list of topics to the user. This leads to the need to cluster web documents based on both the textual and hyperlink information. There exists a large literature on clustering methods and algorithms 13, 19]. Gen- erally speaking, the purpose of cluster analysis is to organize the data into meaningful groups: the data objects in the same group are highly similar and those in di erent groups are dissimilar. Judging the e ectiveness of a clustering algorithm is di cult and usually application-dependent. In this paper, we apply a similarity-based cluster- ing method to the problem of clustering web documents. It utilizes a graph-theoretic criterion called normalized cut which has its root in the study of graph isoperimetric Department of Computer Science and Engineering, The Pennsylvania State University, Uni- versity Park, PA 16802, f xhe,zha g @cse.psu.edu . This work was supported in part by NSF grant CCR-9901986. y NERSC Division, Lawrence Berkeley National Laboratory, University of California, Berkeley, CA 94720, f xfhe,chqding,hdsimon g @lbl.gov . Supported by Department of Energy through an LBL LDRD fund.
30201789
Term norm distribution and its effects on Latent Semantic Indexing
Latent Semantic Indexing (LSI) uses the singular value decomposition to reduce noisy dimensions and improve the performance of text retrieval systems. Preliminary results have shown modest improvements in retrieval accuracy and recall, but these have mainly explored small collections. In this paper we investigate text retrieval on a larger document collection (TREC) and focus on distribution of word norm (magnitude). Our results indicate the inadequacy of word representations in LSI space on large collections. We emphasize the query expansion interpretation of LSI and propose an LSI term normalization that achieves better performance on larger collections.
5650583
Bipartite graph partitioning and data clustering
Many data types arising from data mining applications can be modeled as bipartite graphs, examples include terms and documents in a text corpus, customers and purchasing items in market basket analysis and reviewers and movies in a movie recommender system. In this paper, we propose a new data clustering method based on partitioning the underlying bipartite graph. The partition is constructed by minimizing a normalized sum of edge weights between unmatched pairs of vertices of the bipartite graph. We show that an approximate solution to the minimization problem can be obtained by computing a partial singular value decomposition (SVD) of the associated edge weight matrix of the bipartite graph. We point out the connection of our clustering algorithm to correspondence analysis used in multivariate analysis. We also briefly discuss the issue of assigning data objects to multiple clusters. In the experimental results, we apply our clustering algorithm to the problem of document clustering to illustrate its effectiveness and efficiency.