GEN X 2 Generation of XPath Expressions for Unsupervised Web Data Extraction
The Internet is widely used by everyone and it is possible to find almost everything online. However, there is a lot of information of interest that is not easy to access using search engines. This information comes from back end databases and the web pages are generated dynamically after a user's request. These web pages compose the hidden web and are of great importance since the containing information can be used to make applications that compare products, flights, etc. The objective of this work is to be able to extract all the information of interest automatically and don't extract the information that is not of our interest, like ads or "left bars" without any human intervention. There are already many systems that are able to do this, however they all have problems and they are not completely efficient performing the task. This report presents a novel approach to extract and store information in an unsupervised manner, using Genetic Programming and XPath as resources to reach a solution. Our system, called GEN X2, achieved a precision of 74.66% and recall of 76.06%