Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> Abstract: The course “2L690: Hypermedia Structures and Systems” is ta ught entirely throughWorld Wide Web, and offered at six different universities in the Netherlands and Belgium.Different approaches have been taken towards adding adaptivity to the course text. Thispaper reviews these development steps and presents the ﬁnaldesign, which results in adaptivehyperdocumentsthatcanbewritteninstandardHTML3.2, possiblybyusingofftheshelfHTMLeditors. We also present a simple but powerful representation of user (student) knowledge, asused to adapt the link structure and textual contents of the course text. 1. Introduction Many different deﬁnitions of and techniques for adaptive hypermedia exist. A good overview is presented in[Brusilovsky 1996]. In this paper we follow the terminology introduced in that overview to characterize the kindsof adaptation introduced in the course “2L690: Hypermedia S tructures and Systems”. We describe not only theadaptation techniques used for this course, but also compare them with some other initiatives for using adaptivehypertextin courseware, such as a C Programmingcourse[Kay & Kummerfeld1994a, Kay & Kummerfeld1994b]and the ELM-ART Lisp course [Brusilovsky et al. 1996a].World Wide Web was not designed with highly dynamic applications in mind. A typical characteristic ofadaptive hypertext is that during the reading process the presentation of an information item (e.g. a page) may bedifferent each time that item is revisited. The way some WWW browsers deal with their history mechanism makesit difﬁcult to ensure that pages are reloaded each time they are (virtually) modiﬁed on the server. This problemcan be resolved with some browsers (like Netscape Navigator) but on some others (like NCSA Mosaic for X) itcannot. Unfortunately the browsers cannot be blamed for their behavior because the way they implement theirhistory mechanism satisﬁes the requirements set out by the HyperText Transfer Protocol (HTTP) standard.Authoring adaptive hypermedia is also difﬁcult in a WWW environment, because the HyperText MarkupLanguage (HTML) has no provision for “conditional text”. It is not possible to create a single HTML document ofwhich only selected parts are presented to the user, based on some kind of environment variables controlled by anagent that monitors the user’s knowledge state or preferences. The Interbook tool [Brusilovsky et al. 1996b] forinstanceusesconcept-basedindexingtoprovideaccess to(non-adaptive)HTMLpagesfromdynamicallygeneratedindex pages. Displaying index and “content” pages simultan eously is done through frames, a technique introducedby Netscape but which is not available in all browsers, and which is not part of the latest HTML-3.2 standard.Brusilovsky [Brusilovsky 1996] distinguishes two main categories of adaptivity: <s> BIB001 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext resource discovery system called a Focused Crawler. The goal of a focused crawler is to selectively seek out pages that are relevant to a pre-defined set of topics. The topics are specified not using keywords, but using exemplary documents. Rather than collecting and indexing all accessible Web documents to be able to answer all possible ad-hoc queries, a focused crawler analyzes its crawl boundary to find the links that are likely to be most relevant for the crawl, and avoids irrelevant regions of the Web. This leads to significant savings in hardware and network resources, and helps keep the crawl more up-to-date. To achieve such goal-directed crawling, we designed two hypertext mining programs that guide our crawler: a classifier that evaluates the relevance of a hypertext document with respect to the focus topics, and a distiller that identifies hypertext nodes that are great access points to many relevant pages within a few links. We report on extensive focused-crawling experiments using several topics at different levels of specificity. Focused crawling acquires relevant pages steadily while standard crawling quickly loses its way, even though they are started from the same root set. Focused crawling is robust against large perturbations in the starting set of URLs. It discovers largely overlapping sets of resources in spite of these perturbations. It is also capable of exploring out and discovering valuable resources that are dozens of links away from the start set, while carefully pruning the millions of pages that may lie within this same radius. Our anecdotes suggest that focused crawling is very effective for building high-quality collections of Web documents on specific topics, using modest desktop hardware. © 1999 Published by Elsevier Science B.V. All rights reserved. <s> BIB002 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> Maintaining currency of search engine indices by exhaustive crawling is rapidly becoming impossible due to the increasing size and dynamic content of the web. Focused crawlers aim to search only the subset of the web related to a specific category, and offer a potential solution to the currency problem. The major problem in focused crawling is performing appropriate credit assignment to different documents along a crawl path, such that short-term gains are not pursued at the expense of less-obvious crawl paths that ultimately yield larger sets of valuable pages. To address this problem we present a focused crawling algorithm that builds a model for the context within which topically relevant pages occur on the web. This context model can capture typical link hierarchies within which valuable pages occur, as well as model content on documents that frequently cooccur with relevant pages. Our algorithm further leverages the existing capability of large search engines to provide partial reverse crawling capabilities. Our algorithm shows significant performance improvements in crawling efficiency over standard focused crawling. <s> BIB003 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> Due to limited bandwidth, storage, and computational resources, and to the dynamic nature of the Web, search engines cannot index every Web page, and even the covered portion of the Web cannot be monitored continuously for changes. Therefore it is essential to develop effective crawling strategies to prioritize the pages to be indexed. The issue is even more important for topic-specific search engines, where crawlers must make additional decisions based on the relevance of visited pages. However, it is difficult to evaluate alternative crawling strategies because relevant sets are unknown and the search space is changing. We propose three different methods to evaluate crawling strategies. We apply the proposed metrics to compare three topic-driven crawling algorithms based on similarity ranking, link analysis, and adaptive agents. <s> BIB004 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> Topical crawlers are becoming important tools to support applications such as specialized Web portals, online searching, and competitive intelligence. As the Web mining field matures, the disparate crawling strategies proposed in the literature will have to be evaluated and compared on common tasks through well-defined performance measures. This paper presents a general framework to evaluate topical crawlers. We identify a class of tasks that model crawling applications of different nature and difficulty. We then introduce a set of performance measures for fair comparative evaluations of crawlers along several dimensions including generalized notions of precision, recall, and efficiency that are appropriate and practical for the Web. The framework relies on independent relevance judgements compiled by human editors and available from public directories. Two sources of evidence are proposed to assess crawled pages, capturing different relevance criteria. Finally we introduce a set of topic characterizations to analyze the variability in crawling effectiveness across topics. The proposed evaluation framework synthesizes a number of methodologies in the topical crawlers literature and many lessons learned from several studies conducted by our group. The general framework is described in detail and then illustrated in practice by a case study that evaluates four public crawling algorithms. We found that the proposed framework is effective at evaluating, comparing, differentiating and interpreting the performance of the four crawlers. For example, we found the IS crawler to be most sensitive to the popularity of topics. <s> BIB005 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> Focused crawlers are considered as a promising way to tackle the scalability problem of topic-oriented or personalized search engines. To design a focused crawler, the choice of strategy for prioritizing unvisited URLs is crucial. In this paper, we propose a method using a decision tree on anchor texts of hyperlinks. We conducted experiments on the real data sets of four Japanese universities and verified our approach. <s> BIB006 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines, A focused crawler aims at selectively seek out pages that are relevant to a pre-defined set of topics. Besides specifying topics by some keywords, it is customary also to use some exemplary documents to compute the similarity of a given Web document to the topic, in this paper we introduce a new hybride focused crawler, which uses link structure of documents as well as similarity of pages to the topic to crawl the Web <s> BIB007 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> Focused crawlers aim to search only the subset of the web related to a specific topic, and offer a potential solution to the problem. The major problem is how to retrieve the maximal set of relevant and quality pages. In this paper, We propose an architecture that concentrates more over page selection policy and page revisit policy The three-step algorithm for page refreshment serves the purpose. The first layer contributes to decision of page relevance using two methods. The second layer checks for whether the structure of a web page has been changed or not, the text content has been altered or whether an image is changed. Also a minor variation to the method of prioritizing URLs on the basis of forward link count has been discussed to accommodate the purpose of frequency of update. And finally, the third layer helps to update the URL repository. <s> BIB008 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> As the size of the Web continues to grow, searching it for useful information has become more difficult. Focused crawler intends to explore the Web conform to a specific topic. This paper discusses the problems caused by local searching algorithms. Crawler can be trapped within a limited Web community and overlook suitable Web pages outside its track. A genetic algorithm as a global searching algorithm is modified to address the problems. The genetic algorithm is used to optimize Web crawling and to select more suitable Web pages to be fetched by the crawler. Several evaluation experiments are conducted to examine the effectiveness of the approach. The crawler delivers collections consist of 3396 Web pages from 5390 links which had been visited, or filtering rate of Roulette-Wheel selection at 63% and precision level at 93% in 5 different categories. The result showed that the utilization of genetic algorithm had empowered focused crawler to traverse the Web comprehensively, despite it relatively small collections. Furthermore, it brought up a great potential for building an exemplary collections compared to traditional focused crawling methods. <s> BIB009 </s> Change Detection and Notification of Webpages: A Survey <s> Focused Crawling. <s> A novel design of the focused crawler based on the genetic and ant algorithms is proposed in this paper. The genetic and ant algorithms are combined together to improve the performance of focused crawler. The selection operator, crossover and mutation operator are optimized. The whole improved frame is funded on the new URL analysis model. And experimental results show the new algorithm's efficiency. <s> BIB010
Focused crawling is used to collect pages and documents that belong to a specific topic. This can reduce the workload and network traffic during download as the required pages are only downloaded. It crawls only the relevant regions of the Web. This method saves hardware and network resources significantly. Initially, focused crawling was introduced in BIB002 as "a new approach to topic-specific web resource discovery". The focused crawler has three major components. They are as follows. (1) "Classifier decides whether to expand links on the webpages crawled". (2) "Distiller determines visit priorities of crawled pages". (3) "Crawler consists of dynamically reconfigurable priority controls which is controlled by the classifier and distiller" BIB002 ]. The integration of main components of the focused crawler is depicted in Figure 8 . Fig. 8 . Component integration of the focused crawler BIB002 ]. BIB003 ] has highlighted the importance of assigning credits to different documents across crawling paths that produce larger sets of topically relevant pages. The authors have proposed a focused crawling algorithm with a model that captures link hierarchies containing relevant pages. Later on, BIB004 discussed different ways to compare topic-driven crawling techniques, whereas "a general framework to evaluate topical crawlers" was presented by BIB005 ]. [Pant and Menczer 2003 ] introduced a topical crawling technique to gather documents related to business intelligence, which can support organizations to identify competitors, partners or acquisitions. The authors have tested four crawlers; Breadth-First, Naive Best-First, DOM and Hub-Seeking. The results of precision and recall on a given topic show that the Hub-Seeking crawler outperforms other crawlers. A popular technique to design focused crawlers is the utilization of the "link structure" of the documents. BIB006 has proposed a focused crawling technique using a decision tree created by anchor texts found in hyperlinks. The work done in BIB007 ] has presented a novel "hybrid focused crawler", which utilizes the "link structure" and similarity of the content in webpages to a particular topic. [ BIB008 has proposed another approach for a web crawler with focused crawling features. Three layers are present in this architecture; "page relevance computation", "determination of page change" and "updating the URL repository" BIB008 . During the crawling mechanism, all the pages are not downloaded, instead it extracts the URLs and the words of interest. Frequency of related words and the number of forward and backward links to and from the webpage collaboratively decides the importance of that webpage being parsed. Certain parameters such as topic vectors and relevance scores are used to check the importance of the page. If the relevance level exceeds a predefined threshold, it is downloaded for extraction. Otherwise the page will be discarded. Work done in  has studied focused web crawlers with their advantages and disadvantages. Additionally, the authors have suggested methods to further improve the efficiency of web crawlers. With the advancement in optimization algorithms, researchers have turned their focus on optimizing web crawlers. Optimizations allow web crawlers to select more suitable web pages to be fetched. Focused crawlers have been optimized to increase the efficiency and performance of the crawler using search algorithms such as "Breadth-First Search"  , "Fish-Search" BIB001 , "Shark-Search"  ] and evolutionary algorithms such as "Genetic algorithms" BIB009 ] BIB010 and "Ant algorithms" BIB010 ].