CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(a) and37CFR § 1.55 to UK Patent Application No. GB1804921.3, filed on Mar. 27, 2018, the entire content of which is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to web crawling; and more specifically, to systems that crawls a wide area computer network for retrieving contextual information. Furthermore, the present disclosure relates to methods for crawling a wide area computer network for retrieving contextual information. Moreover, the present disclosure also relates to computer readable medium containing program instructions for execution on a computer system, which when executed by a computer, cause the computer to perform method steps of crawling a wide area computer network for retrieving contextual information.

BACKGROUND

In recent years, there has been tremendous increase in use of computing devices, such as personal computers, laptops, connected to a global network of millions of other computers, namely the World Wide Web. Furthermore, the computing devices (such as servers) are operable to store digital information, and allow access to digital information stored therein. Furthermore, in such set-up the computing devices are connected to other computers in the World Wide Web network and capable of accessing digital information therefrom. Moreover, with development of technology, accessing the digital information has been automated by introducing programs that are operable to access (crawl) the computers (such as servers storing websites, webpages and the likes) of the World Wide Web to download and upload digital information. The programs use Uniform Resource Identifier (URI) associated with the servers to download and upload digital information.

However, such conventional setups for crawling the World Wide Web includes a number of problems. Typically, in conventional crawling arrangements, a fixed infrastructure, including a large number of resources, is used for browsing the World Wide Web. Such conventional crawling arrangements have an inherent problem of not being able to scale the number of resources for accessing the World Wide Web depending upon requirement thereof. Therefore, such conventional crawling arrangements are only suited to access a fixed number of URIs. Furthermore, the conventional crawling arrangements can be time consuming as the conventional crawling arrangements crawl URIs irrespective of relevancy thereof. Additionally, at an instance of unexpected increase in number of URIs to be crawled, the conventional crawling arrangements get overloaded. Consequently, such overloading may lead to hardware failure, high maintenance and thereby increasing the operation cost. Moreover, the conventional crawling arrangements generate crawling output data in restricted data formats, therefore the output data needs to be further processed, cleaned and converted to required data format manually. Thus, the conventional crawling arrangements are non-optimized, time consuming and cumbersome to operate.

Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with conventional crawling of the World Wide Web

SUMMARY

The present disclosure seeks to provide a system that crawls a wide area computer network for retrieving contextual information. The present disclosure also seeks to provide a method of crawling a wide area computer network for retrieving contextual information. The present disclosure also seeks to provide a computer readable medium, containing program instructions for execution on a computer system, which when executed by a computer, cause the computer to perform method steps of crawling a wide area computer network for retrieving contextual information. The present disclosure seeks to provide a solution to the existing problem of inefficient and time-consuming web crawling. Additionally, the present disclosure seeks to provide a solution to optimize efficiency and performance of hardware system that performs the aforementioned method. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in prior art, and provide an optimized, faster and efficient method of web crawling and extracting relevant information from the web.

In one aspect, an embodiment of the present disclosure provides a system that crawls a wide area computer network for retrieving contextual information, wherein the system includes a computer system, wherein the system comprises:a data processing arrangement comprising a communication interface for accessing the wide area computer network and a web crawling module, wherein the web crawling module is operable to:receive an input file via a user interface provided by the data processing arrangement, wherein the input file includes a plurality of Uniform Resource Identifiers, wherein each of the plurality of Uniform Resource Identifiers points to a corresponding web location in the wide area computer network;identify, from the plurality of Uniform Resource Identifiers, accessible Uniform Resource Identifiers based on one or more parameters associated with each of the plurality of Uniform Resource Identifiers;obtain one or more features associated with the accessible Uniform Resource Identifiers; andexecute an event for crawling the accessible Uniform Resource Identifiers to extract web content available in the corresponding web locations, wherein executing the event comprises:creating at least one resource cluster for executing the event, wherein the at least one resource cluster comprises one or more resources acquired based on the one or more features associated with the accessible Uniform Resource Identifiers;acquiring at least one electronic record file to be executed by the at least one resource cluster, wherein acquiring the at least one electronic record file is based on the one or more features associated with the accessible Uniform Resource Identifiers; andcrawling the accessible Uniform Resource Identifiers using the at least one resource cluster; andprocess the extracted web content to identify contextual data, wherein the contextual data is organized into one or more data structure; anda database arrangement communicably coupled to the data processing arrangement via the communication interface, wherein the database arrangement is operable to aggregate the contextual data into one or more database arranged therein.

In another aspect, an embodiment of the present disclosure provides a method of crawling a wide area computer network for retrieving contextual information, wherein the method includes using a computer system, wherein the method comprises:receiving an input file, wherein the input file includes a plurality of Uniform Resource Identifiers, wherein each of the plurality of Uniform Resource Identifiers points to a corresponding web location in the wide area computer network;identifying, from the plurality of Uniform Resource Identifiers, accessible Uniform Resource Identifiers based on one or more parameters associated with each of the plurality of Uniform Resource Identifiers;obtaining one or more features associated with the accessible Uniform Resource Identifiers;executing an event for crawling the accessible Uniform Resource Identifiers to extract web content available in the corresponding web locations, wherein executing the event comprises:creating at least one resource cluster for executing the event, wherein the at least one resource cluster comprises one or more resources acquired based on the one or more features associated with the accessible Uniform Resource Identifiers;acquiring at least one electronic record file to be executed by the at least one resource cluster, wherein acquiring at least one electronic record file is based on the one or more features associated with the accessible Uniform Resource Identifiers; andcrawling the accessible Uniform Resource Identifiers using the at least one resource cluster;processing the extracted web content to identify contextual data, wherein the contextual data is organized into one or more data structure; andaggregating the contextual data.

In yet another aspect, an embodiment of the present disclosure provides a computer readable medium, containing program instructions for execution on a computer system, which when executed by a computer, cause the computer to perform method steps for identifying at least one nascent topic related to a subject matter, the method comprising the steps of:receiving an input file, wherein the input file includes a plurality of Uniform Resource Identifiers, wherein each of the plurality of Uniform Resource Identifiers points to a corresponding web location in the wide area computer network;identifying, from the plurality of Uniform Resource Identifiers, accessible Uniform Resource Identifiers based on one or more parameters associated with each of the plurality of accessible Uniform Resource Identifiers;obtaining one or more features associated with the accessible Uniform Resource Identifiers;executing an event for crawling the accessible Uniform Resource Identifiers to extract web content available in the corresponding web locations, wherein executing the event comprises:creating at least one resource cluster for executing the event, wherein the at least one resource cluster comprises one or more resources acquired based on the one or more features associated with the accessible Uniform Resource Identifiers;acquiring at least one electronic record file to be executed by the at least one resource cluster, wherein acquiring at least one electronic record file is based on the one or more features associated with the accessible Uniform Resource Identifiers; andcrawling the accessible Uniform Resource Identifiers using the at least one resource cluster;processing the extracted web content to identify contextual data, wherein the contextual data is organized into one or more data structure; andaggregating the contextual data.

Embodiments of the present disclosure substantially eliminate or at least partially address the aforementioned problems in the prior art, and enables an efficient and reliable method for performing web crawling.

Additional aspects, advantages, features and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative embodiments construed in conjunction with the appended claims that follow.

It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

