Change Detection and Notification of Webpages: A Survey <s> DETECTING CHANGES OF WEBPAGES Web Crawlers and Crawling Techniques <s> In this paper we present an analysis of an AltaVista Search Engine query log consisting of approximately 1 billion entries for search requests over a period of six weeks. This represents almost 285 million user sessions, each an attempt to fill a single information need. We present an analysis of individual queries, query duplication, and query sessions. We also present results of a correlation analysis of the log entries, studying the interaction of terms within queries. Our data supports the conjecture that web users differ significantly from the user assumed in the standard information retrieval literature. Specifically, we show that web users type in short queries, mostly look at the first 10 results only, and seldom modify the query. This suggests that traditional information retrieval techniques may not work well for answering web search requests. The correlation analysis showed that the most highly correlated items are constituents of phrases. This result indicates it may be useful for search engines to consider search terms as parts of phrases even if the user did not explicitly specify them as such. <s> BIB001 </s> Change Detection and Notification of Webpages: A Survey <s> DETECTING CHANGES OF WEBPAGES Web Crawlers and Crawling Techniques <s> Information Retrieval deals with searching and retrieving information within the documents and it also searches the online databases and internet. Web crawler is defined as a program or software which traverses the Web and downloads web documents in a methodical, automated manner. Based on the type of knowledge, web crawler is usually divided in three types of crawling techniques: General Purpose Crawling, Focused crawling and Distributed Crawling. In this paper, the applicability of Web Crawler in the field of web search and a review on Web Crawler to different problem domains in web search is discussed. <s> BIB002 </s> Change Detection and Notification of Webpages: A Survey <s> DETECTING CHANGES OF WEBPAGES Web Crawlers and Crawling Techniques <s> Although web crawlers have been around for twenty years by now, there is virtually no freely available, open-source crawling software that guarantees high throughput, overcomes the limits of single-machine systems, and, at the same time, scales linearly with the amount of resources available. This article aims at filling this gap, through the description of BUbiNG, our next-generation web crawler built upon the authors’ experience with UbiCrawler [9] and on the last ten years of research on the topic. BUbiNG is an open-source Java fully distributed crawler; a single BUbiNG agent, using sizeable hardware, can crawl several thousand pages per second respecting strict politeness constraints, both host- and IP-based. Unlike existing open-source distributed crawlers that rely on batch techniques (like MapReduce), BUbiNG job distribution is based on modern high-speed protocols to achieve very high throughput. <s> BIB003
"A web crawler (also known as a spider) is a software or a programmed script that browses the WWW in a systematic, automated manner" BIB002 ] and systematically downloads numerous webpages starting from a seed BIB003 ]. Web crawlers date back to the 1990s, where they were introduced when the WWW was invented. The "World Wide Web Worm"  ] and "MOMspider"  ] were among the early web crawlers. Moreover, commercial search engines developed their own web crawlers as well such as "Google crawler" ], "AltaVista" BIB001 ], etc. Later on, web crawlers which could efficiently download millions of webpages were built. We can consider the Internet to be a "directed graph" where each node represents a webpage and the edges represent hyperlinks connecting these webpages BIB002 ]. Web crawlers traverse over this graph like structure of the Internet, go to webpages and download their content for indexing purposes. It is crucial to identify which crawling technique should be used according to the purpose of the application. Web crawling can be considered as the main process behind web caching, search engines and web information retrieval. A web crawler begins crawling from a seed URL and visit pages. Then it downloads the page and retrieves the URLs in it. The discovered URLs will be kept in a queue and this process repeats as the crawler travels from page to page. Figure 7 shows an overview of the web crawling process. Many crawling techniques are being used by web crawlers at present BIB002 ] such as (1) "General Purpose Crawling", (2) "Focused Crawling" and (3) "Distributed Crawling" BIB002 ]. 4.1.1. General Purpose Crawling. In the general-purpose crawling technique, the web crawlers collect all the possible webpages from a set of URLs and their links to a central location. It can fetch numerous pages from many different locations. However, this technique can slow down the crawling process and reduce the network bandwidth as all the pages are being fetched.