Change Detection and Notification of Webpages: A Survey <s> Detecting the Dynamic Nature of Webpages <s> Changes in the topography of the Web can be expressed in at least four ways: (1) more sites on more servers in more places, (2) more pages and objects added to existing sites and pages, (3) changes in traffic, and (4) modifications to existing text, graphic, and other Web objects. This article does not address the first three factors (more sites, more pages, more traffic) in the growth of the Web. It focuses instead on changes to an existing set of Web documents. The article documents changes to an aging set of Web pages, first identified and collected in December 1996 and followed weekly thereafter. Results are reported through February 2001. The article addresses two related phenomena: (1) the life cycle of Web objects, and (2) changes to Web objects. These data reaffirm that the half-life of a Web page is approximately 2years. There is variation among Web pages by top-level domain and by page type (navigation, content). Web page content appears to stabilize over time; aging pages change less often than once they did. <s> BIB001 </s> Change Detection and Notification of Webpages: A Survey <s> Detecting the Dynamic Nature of Webpages <s> How fast does the web change? Does most of the content remain unchanged once it has been authored, or are the documents continuously updated? Do pages change a little or a lot? Is the extent of change correlated to any other property of the page? All of these questions are of interest to those who mine the web, including all the popular search engines, but few studies have been performed to date to answer them.One notable exception is a study by Cho and Garcia-Molina, who crawled a set of 720,000 pages on a daily basis over four months, and counted pages as having changed if their MD5 checksum changed. They found that 40% of all web pages in their set changed within a week, and 23% of those pages that fell into the .com domain changed daily.This paper expands on Cho and Garcia-Molina's study, both in terms of coverage and in terms of sensitivity to change. We crawled a set of 150,836,209 HTML pages once every week, over a span of 11 weeks. For each page, we recorded a checksum of the page, and a feature vector of the words on the page, plus various other data such as the page length, the HTTP status code, etc. Moreover, we pseudo-randomly selected 0.1% of all of our URLs, and saved the full text of each download of the corresponding pages.After completion of the crawl, we analyzed the degree of change of each page, and investigated which factors are correlated with change intensity. We found that the average degree of change varies widely across top-level domains, and that larger pages change more often and more severely than smaller ones.This paper describes the crawl and the data transformations we performed on the logs, and presents some statistical observations on the degree of change of different classes of pages. <s> BIB002 </s> Change Detection and Notification of Webpages: A Survey <s> Detecting the Dynamic Nature of Webpages <s> It is crucial for a web crawler to distinguish between ephemeral and persistent content. Ephemeral content (e.g., quote of the day) is usually not worth crawling, because by the time it reaches the index it is no longer representative of the web page from which it was acquired. On the other hand, content that persists across multiple page updates (e.g., recent blog postings) may be worth acquiring, because it matches the page's true content for a sustained period of time. In this paper we characterize the longevity of information found on the web, via both empirical measurements and a generative model that coincides with these measurements. We then develop new recrawl scheduling policies that take longevity into account. As we show via experiments over real web data, our policies obtain better freshness at lower cost, compared with previous approaches. <s> BIB003 </s> Change Detection and Notification of Webpages: A Survey <s> Detecting the Dynamic Nature of Webpages <s> The Web is a dynamic, ever changing collection of information. This paper explores changes in Web content by analyzing a crawl of 55,000 Web pages, selected to represent different user visitation patterns. Although change over long intervals has been explored on random (and potentially unvisited) samples of Web pages, little is known about the nature of finer grained changes to pages that are actively consumed by users, such as those in our sample. We describe algorithms, analyses, and models for characterizing changes in Web content, focusing on both time (by using hourly and sub-hourly crawls) and structure (by looking at page-, DOM-, and term-level changes). Change rates are higher in our behavior-based sample than found in previous work on randomly sampled pages, with a large portion of pages changing more than hourly. Detailed content and structure analyses identify stable and dynamic content within each page. The understanding of Web change we develop in this paper has implications for tools designed to help people interact with dynamic Web content, such as search engines, advertising, and Web browsers. <s> BIB004 </s> Change Detection and Notification of Webpages: A Survey <s> Detecting the Dynamic Nature of Webpages <s> Many web documents are dynamic, with content changing in varying amounts at varying frequencies. However, current document search algorithms have a static view of the document content, with only a single version of the document in the index at any point in time. In this paper, we present the first published analysis of using the temporal dynamics of document content to improve relevance ranking. We show that there is a strong relationship between the amount and frequency of content change and relevance. We develop a novel probabilistic document ranking algorithm that allows differential weighting of terms based on their temporal characteristics. By leveraging such content dynamics we show significant performance improvements for navigational queries. <s> BIB005
Identifying the amount of change and changing patterns of webpages has been of great interest to researchers. Many studies have been done to understand the changing nature of webpages. The content of webpages and their change frequencies have been highly focused areas in this research scope. The available literature demonstrates the ever-changing nature of webpages and the various aspects that were reasons for those changes. Different factors have been considered regarding the dynamic behavior of web content. have conducted an experiment with 720,000 webpages from a number of 270 websites to study how web pages evolve over time. They have crawled these web pages every day for 4 months. Based on the results, the researchers have found that more than 20% of the webpages had shown changes whenever they visited them and 40% of the .com webpages had changed every day. Over 50% of the .com and .edu webpages had not change at all during the experiment's time frame. A massive scale experiment which extended the study done by Cho and Garcia-Molina, was performed in BIB002 ]. They have studied how frequently webpages change and the quantity of change occurred, using approximately 150 million webpages over 11 weeks of time period. According to their findings, approximately 65% of the pages had zero change over the considered time period. Further, it shows the relationships in-between certain top level domain types (e.g.: .org, .gov, .com) and their respective frequencies of changing. It was revealed that .org and .gov domains are less likely to change than .gov domains and pages which include spams would change more frequently. Work done in BIB001 ] presents a study on webpages carried out over four years to understand the changing aspects of webpages as they age. The findings show that a webpage has a "half-life" of about 2-years. The half-life is the "time period required for half of a defined web literature to disappear"  ]. In Kohler's continued work  , it is concluded that once a website has sufficiently aged, the webpages tend to stabilize. Moreover, [Ntoulas et al. 2004 ] explored 150 websites by taking weekly snapshots for a period of one year. Even though webpages changed, they changed slightly, and the changes did not follow the similarity measure of "bagof-words" vector similarity. The authors have stated that the degree of change could be considered a good factor in forecasting changes which could happen in the future. Another interesting work has been carried out in BIB003 . The authors have crawled 10,000 web URLs from Yahoo crawled collection and 10,000 web URLs from the Open Directory listing. A webpage is modeled as a collection of regions where content inside one region has no dependency over other and each region has one of the three behaviors; "Static behaviour" where the page content remains unchanged, "Scroll behaviour" where the content is added to the page periodically and "Churn behaviour" where the page content is overwritten. According to the results, from the dynamic content on webpages from the Open Directory listing, about a third has showed a scroll behavior. BIB004 ] have crawled and analyzed approximately 55,000 webpages, which are revisited by users, to characterize the various changes occurring in them. The authors have tracked the frequency and amount of change that has occurred in the webpages individually. 34% of the webpages had not shown any changes whereas the remaining pages had displayed at least one change for every 123 hours. This study has shown that popular webpages change more frequently considered to less popular webpages. Webpages falling under categories such as sports, news and personal pages change most frequently and webpages with government and educational domain addresses has no frequent changes. However, with the currently growing popularity of educational and government websites, they have become more dynamic in nature and their content gets updated frequently. Furthermore, the work carried out in BIB005 describes the temporal aspects of changing web documents. The authors describe news websites to consist of highly changing webpages. Whenever new stories are available, or the old stories are updated, the news websites would change. These changes occur in different amounts and at different frequencies. In order to observe how documents are changed, the authors created a set of approximately 2,000,000 HTML pages and crawled them in weekly basis for 10 weeks. Over the sampled time period, over 62% of pages had no virtual difference. They have also pointed out that highly relevant documents are both more likely to change and contain a higher amount of change than general documents. As the final outcome, the authors have proposed a ranked retrieval model for dynamic documents based on temporal aspect of content which lets differential weighting of content. Work done in [Saad and and Gancarski 2012] has monitored over 100 webpages from the France Televisions (FranceTV) archive, which depicts the evolution of changes within the French Web. Each page was crawled every hour and over 3000 versions were obtained daily. The results have shown that the pages at the root level of the archive, such as homepages changed significantly, whereas pages in the deepest levels did not change.