The Word Wide Web enables a web resource, such as a document, to be identified by a web address, such as a Uniform Resource Locator (URL) that provides a global naming scheme for web resources. A web address specifies a web resource's location on the World Wide Web and explains the means by which a web resource can be accessed via the Internet. For example, a URL for a web page may utilize Hypertext Transfer Protocol (HTTP), and when the URL is entered into an address bar of a web browser, then the web browser displays the web page. The web page may include information including a hypertext link, such as Hypertext Markup Language (HTML), that provides for the web page to link to other web pages. It is these hyperlinks of web pages that create what can be called a web of information, which is another description of the World Wide Web.
Although it is not possible to know how many web sites there are on the World Wide Web, an estimate is that the count is above one billion. Such reflects an immense amount of information for a user to manually browse with a web browser, such as Microsoft Internet Explorer, Google Chrome, and the like. In order to assist users in finding and processing information available on the World Wide Web, a process known as web crawling has been developed to automate the browsing, grabbing and sifting of information on the World Wide Web. Such can attempt to simulate a user's exploration of the World Wide Web to automatically extract useful information. Web crawling is accomplished with a web robot that is a software application of process instructions that execute automated and repetitive tasks via the Internet. One of those repetitive tasks is to extract structured and unstructured information from a web page, and convert that information into a form that can be stored in a central database. Such data extraction from a web page is commonly called web scraping, and may be beneficial for online price comparisons, centralized job postings on career web sites, and analyzing market research, among other uses. Web scraping allows for the collection of information from a web site without having to directly access any back-end databases, for example, through applicable application programming interfaces (APIs). Traditional financial management systems provide a mechanism to upload information from a service provider's system to the financial management system, such as through a data transfer interface that may utilize APIs. For example, a banking system may transfer a user's bank transactional records to a financial management system via APIs associated with the Open Financial Exchange (OFX) Standard or the like.
Web sites are generally intended to be displayed to an end-user and are not generally intended to be utilized for data transfers between software systems. Nevertheless, scraping is a data transfer process that can be at a semantic level or a syntactic level. Semantic level scraping utilizes a common framework for structured data. A web page may utilize a markup language such as Extensible Markup Language (XML) that promotes structured data with rules for encoding data that is both human-readable and machine-readable. A script may be created from a scripting language such as PERL, Tcl, Python, and the like, and a script may extract well-formed XML content comprising data at the semantic level. Under some embodiments, a script comprises processing instructions. Such may be advantageous because a web page's HTML may change due to changes in the look and feel of the web page, yet the script can still extract the XML content at a semantic level. For example, the data may be presented in a systematic way through XML elements that convey the meaning of the data.
On the other hand, syntactic level scraping refers to scraping data that is not presented in a systematic way. Web scraping at the syntactic level is often considered a manually intensive process to create scripts to extract information. For example, one may rely on knowledge of specific markup practices or document structures to determine data items among other web page elements. Such knowledge can be used to recognize data structures on a web page, and create a script to extract data at the syntactic level. Scraping data that is presented in a syntactic way can be prone to difficulties because when a web page's layout or HTML changes, the script must often be rewritten in response to continue to scrape the web page at the syntactic level. Until the script is rewritten, the data that had been retrieved in the past can often no longer be retrieved.
The creation of a script to scrape a web page at the syntactical level may be further complicated by the lack of domain-specific knowledge for a web page, in which a domain may be an industry grouping such as financial institutions, healthcare institutions, insurance institutions, travel institutions, legal institutions, and others. Having knowledge to create scraping scripts in one domain does not necessarily mean that one can create scraping scripts in other domains. For example, the creation of a script to scrape a financial institution web page at the syntactic level may require knowledge about how financial transactions are presented on a web page. In contrast, the creation of a script to scrape a healthcare institution web page at the syntactic level may require knowledge about how healthcare transactions are presented on a web page. Because of the diversity of domains, a developer who creates scripts and who possesses the knowledge to extract transactions from one domain may not have the knowledge to extract transactions from another domain.
Accordingly, there is a need to improve the creation of scraping scripts that extract information from web pages across different domains.