Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> For life testing procedures, a Bayesian analysis is developed with respect to a random intensity parameter in the Poisson distribution. Bayes estimators are derived for the Poisson parameter and the reliability function based on uniform and gamma prior distributions of that parameter. A Monte Carlo procedure is implemented to make possible an empirical mean-squared error comparison between Bayes and existing minimum variance unbiased, as well as maximum likelihood, estimators. As expected, the Bayes estimators have mean-squared errors that are appreciably smaller than those of the other two. <s> BIB001 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> A robot is deployed by a Web search engine in order to maintain the currency of its data base of Web pages. This paper studies robot scheduling policies that minimize the fractions $r_i$ of time pages spend out-of-date, assuming independent Poisson page-change processes, and a general distribution for the page access time $X$. We show that, if $X$ is decreased in the increasing-convex ordering sense, then $r_i$ is decreased for all $i$ under any scheduling policy, and that, in order to minimize expected total obsolescence time of any page, the accesses to that page should be as evenly spaced in time as possible. We then investigate the problem of scheduling to minimize the cost function $\sum c_i r_i,$ where the $c_i$ are given weights proportional to the page-change rates $\mu_i$. We give a tight bound on the performance of such a policy and prove that the optimal frequency at which the robot should access page $i$ is proportional to $\ln (h_i)^{-1}$, where $h_i := {\rm E}e^{-\mu_iX}.$ Note that this reduces to being proportional to $\mu_i$ when $X$ is a constant, but not, as one might expect, when $X$ has a general distribution. Next, we evaluate randomized accessing policies whereby the choices of page access are determined by independent random samples from the distribution ${f_i}$. We show that when the weights $c_i$ in the cost function are proportional to $\mu_i$, the minimum cost is achieved when $f_i$ is proportional to $(h_i)^{-1} - 1$. Finally, we present and analyze a heuristic policy that is especially suited to the asymptotic regime of large data bases. <s> BIB002 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> How fast does the web change? Does most of the content remain unchanged once it has been authored, or are the documents continuously updated? Do pages change a little or a lot? Is the extent of change correlated to any other property of the page? All of these questions are of interest to those who mine the web, including all the popular search engines, but few studies have been performed to date to answer them.One notable exception is a study by Cho and Garcia-Molina, who crawled a set of 720,000 pages on a daily basis over four months, and counted pages as having changed if their MD5 checksum changed. They found that 40% of all web pages in their set changed within a week, and 23% of those pages that fell into the .com domain changed daily.This paper expands on Cho and Garcia-Molina's study, both in terms of coverage and in terms of sensitivity to change. We crawled a set of 150,836,209 HTML pages once every week, over a span of 11 weeks. For each page, we recorded a checksum of the page, and a feature vector of the words on the page, plus various other data such as the page length, the HTTP status code, etc. Moreover, we pseudo-randomly selected 0.1% of all of our URLs, and saved the full text of each download of the corresponding pages.After completion of the crawl, we analyzed the degree of change of each page, and investigated which factors are correlated with change intensity. We found that the average degree of change varies widely across top-level domains, and that larger pages change more often and more severely than smaller ones.This paper describes the crawl and the data transformations we performed on the logs, and presents some statistical observations on the degree of change of different classes of pages. <s> BIB003 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> Many online data sources are updated autonomously and independently. In this article, we make the case for estimating the change frequency of data to improve Web crawlers, Web caches and to help data mining. We first identify various scenarios, where different applications have different requirements on the accuracy of the estimated frequency. Then we develop several "frequency estimators" for the identified scenarios, showing analytically and experimentally how precise they are. In many cases, our proposed estimators predict change frequencies much more accurately and improve the effectiveness of applications. For example, a Web crawler could achieve 35p improvement in "freshness" simply by adopting our proposed estimator. <s> BIB004 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> It is crucial for a web crawler to distinguish between ephemeral and persistent content. Ephemeral content (e.g., quote of the day) is usually not worth crawling, because by the time it reaches the index it is no longer representative of the web page from which it was acquired. On the other hand, content that persists across multiple page updates (e.g., recent blog postings) may be worth acquiring, because it matches the page's true content for a sustained period of time. In this paper we characterize the longevity of information found on the web, via both empirical measurements and a generative model that coincides with these measurements. We then develop new recrawl scheduling policies that take longevity into account. As we show via experiments over real web data, our policies obtain better freshness at lower cost, compared with previous approaches. <s> BIB005 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> This is a survey of the science and practice of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures to theoretical questions such as how often to revisit evolving content sources. This survey outlines the fundamental challenges and describes the state-of-the-art models and solutions. It also highlights avenues for future work. <s> BIB006 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> The Internet at present has become vast and dynamic with the ever increasing number of web pages. These web pages change when more content is added to them. With the availability of change detection and notification systems, keeping track of the changes occurring in web pages has become more simple and straightforward. However, most of these change detection and notification systems work based on predefined crawling schedules with static time intervals. This can become inefficient if there are no relevant changes being made to the web pages, resulting in the wastage of both temporal and computational resources. If the web pages are not crawled frequently, some of the important changes may be missed and there may be delays in notifying the subscribed users. This paper proposes a methodology to detect the frequency of change in web pages to optimize server-side scheduling of change detection and notification systems. The proposed method is based on a dynamic detection process, where the crawling schedule will be adjusted accordingly in order to result in a more efficient server-based scheduler to detect changes in web pages. <s> BIB007 </s> Change Detection and Notification of Webpages: A Survey <s> Frequency of Webpage Changes <s> The backbone of every search engine is the set of web crawlers, which go through all indexed web pages and update the search indexes with fresh copies, if there are changes. The crawling process provides optimum search results by keeping the indexes refreshed and up to date. This requires an "ideal scheduler" to crawl each web page immediately after a change occurs. Creating an optimum scheduler is possible when the web crawler has information about how often a particular change occurs. This paper discusses a novel methodology to determine the change frequency of a web page using machine learning and server scheduling techniques. The methodology has been evaluated with 3000+ web pages with various changing patterns. The results indicate how Information Access (IA) and Performance Gain (PG) are balanced out to zero in order to create an optimum crawling schedule for search engine indexing. <s> BIB008
Since the 1970s, several studies have been carried out based on statistics, to estimate the change frequency  BIB001 ]. Nevertheless, majority of these studies have been done under the assumption that the history of changes is available for the webpage, which might not be true always in the area of change detection of webpages. When analyzing CDN systems, it is visible that the complete change histories will not be available for a webpage being tracked. In several related studies BIB003 ]  BIB005 , the "Poisson model" has been introduced as a model which can be used to estimate rate of change of a webpage (also known as change frequency). Most of the work which has been carried out are with an assumption: "changes arrive as a Poisson process, and that the average rate of change can be estimated under this model." According to BIB006 , the "binary freshness model" can be used to measure the freshness in a webpage. This model which is also known as obsolescence, is a function that is of the form f(p, t) {0, 1} (1) where f(p, t) denotes "the freshness" of a particular webpage p over a t time period. It compares the live copy of a specific webpage p with a cached version of the webpage across a time period t to check if they are identical (or near-identical). Under this model, if f(p, t) equals to one, then the webpage p can be called "fresh" over the time t, whereas otherwise it is coined as a "stale" webpage which has gone through some change. This is a simple, but an effective model which provides readers with a very good understanding about webpage changes. The first most study regarding "freshness maximization problem" was in BIB002 , where the authors proposed a Poisson model for webpage changes. A set of random and independent events which occur in a fixed-rate can be modelled using the Poisson Process. A webpage can undergo changes which cause the cached copy of the web crawler go "stale". If λ(p) is the rate parameter of a Poisson distribution where p is the webpage, that specific distribution can be used to denote the occurrence of changes in that specific webpage. This also suggests that the changes happen independently and randomly with a mean rate of λ(p) changes/unit of time. However, this "binary freshness model" lacks the ability to determine whether one page is fresher than the other since the model outputs a binary value; fresh or stale. Hence  has introduced a non-binary freshness model known as the "temporal freshness metric" which is, f(p, t) ∝ age(p, t) (2) in which age(p, t) represents the age of a page denoted by p up to time t and age (p, t) ∈ {0, a} with a as the time duration the copies differed. If a cached-copy of webpage p is indistinguishable from its live-copy, then age(p, t) = 0.The intuition of this is methodology is that "the more time a cached page stays unsynchronized with its livecopy, the more their content tends to drift away". Work done in BIB004 has proposed several frequency estimators for different online applications which require frequency estimation with different accuracy levels. The authors have proposed frequency estimators for scenarios, where the existence of a change is known, and last date of change is known. Furthermore, the authors have proposed a model to categorize elements in to different classes based on their change frequencies. These aspects can be modeled in CDN systems to categorize webpages once their change frequencies have been estimated. In  , the authors have stated that for every one of the webpages, the hourly update rate, can be represented by a Poisson distribution together with λ = 1/Δ where λ is a parameter and Δ is the mean time between changes. The authors have also described a mechanism to identify webpage changes and a model for the computation of the rate of change of a given website. Work carried out in BIB007 ] has proposed a methodology to prepredict the update frequency in webpages (rate at which changes happen) and reschedule the crawling schedule, making the process of crawling webpages more efficient in search engines and web crawlers used for change detection. The authors have introduced a change frequency detection system named as "Change Frequency Estimator Module (CFEM)". It includes two processes. Whenever a fresh webpage is added, that webpage will be crawled to detect a suitable change frequency and it will be recorded in the system. Then these values are sent to a machine learning model BIB008 which will predict the time interval between two crawls for a particular webpage. The machine learning model used to predict a time interval between two crawls for a particular webpage was implemented based on a modified random forest classifier model in H2O.ai machine learning API  [Distributed Random Forest (DRF) -H2O 3.12.0.1 documentation 2017]. For the implementation, the authors have used the time intervals (known as loop-time values) These loop time values are a time interval (in hours) and when the change values together with change frequencies for a webpage is sent to the machine learning model, it would output a loop-time value corresponding to that particular webpage. This value is an estimation of the average time taken by the webpage between 2 changes or in other terms, the refresh rate. It has also been observed by the authors that frequently changing websites obtained lower loop times in comparison to webpages which do not change often. BIB007 