We propose a method to generate large-scale encyclopedic knowledge , which is valuable for much NLP research , based on the Web . We first search the Web for pages containing a term in question . Then we use linguistic patterns and HTML structures to extract text fragments describing the term . Finally , we organize extracted term descriptions based on word senses and domains . In addition , we apply an automatically generated encyclopedia to a question answering system targeting the Japanese Information-Technology Engineers Examination . Reflecting the growth in utilization of the World Wide Web , a number of Web-based language processing methods have been proposed within the natural language processing ( NLP ) , information retrieval ( IR ) and artificial intelligence ( AI ) communities . A sample of these includes methods to extract linguistic resources ( Fujii and Ishikawa , 2000 ; Resnik , 1999 ; Soderland , 1997 ) , retrieve useful information in response to user queries ( Etzioni , 1997 ; McCallum et al . , 1999 ) and mine / discover knowledge latent in the Web ( Inokuchi et al . , 1999 ) . In this paper , mainly from an NLP point of view , we explore a method to produce linguistic resources . Specifically , we enhance the method proposed by Fujii and Ishikawa ( 2000 ) , which extracts encyclopedic knowledge ( i.e. , term descriptions ) from the Web . In brief , their method searches the Web for pages containing a term in question , and uses linguistic expressions and HTML layouts to extract fragments describing the term . They also use a language model to discard non-linguistic fragments . In addition , a clustering method is used to divide descriptions into a specific number of groups . On the one hand , their method is expected to enhance existing encyclopedias , where vocabulary size is relatively limited , and therefore the quantity problems has been resolved . On the other hand , encyclopedias extracted from the Web are not comparable with existing ones in terms of quality . In hand-crafted encyclopedias , term descriptions are carefully organized based on domains and word senses , which are especially effective for human usage . However , the output of Fujii 's method is simply a set of unorganized term descriptions . Although clustering is optionally performed , resultant clusters are not necessarily related to explicit criteria , such as word senses and domains . To sum up , our belief is that by combining extraction and organization methods , we can enhance both quantity and quality of Web-based encyclopedias . Motivated by this background , we introduce an organization model to Fujii 's method and reformalize the whole framework . In other words , our proposed method is not only extraction but generation of encyclopedic knowledge . Section 2 explains the overall design of our encyclopedia generation system , and Section 3 elaborates on our organization model . Section 4 then explores a method for applying our resultant encyclopedia to NLP research , specifically , question answering . Section 5 performs a number of experiments to evaluate our methods . The World Wide Web has been an unprecedentedly enormous information source , from which a number of language processing methods have been explored to extract , retrieve and discover various types of information . In this paper , we aimed at generating encyclopedic knowledge , which is valuable for many applications including human usage and natural language understanding . For this purpose , we reformalized an existing Web-based extraction method , and proposed a new statistical organization model to improve the quality of extracted data . Given a term for which encyclopedic knowledge ( i.e. , descriptions ) is to be generated , our method sequentially performs a ) retrieval of Web pages contain-ing the term , b ) extraction of page fragments describing the term , and c ) organizing extracted descriptions based on domains ( and consequently word senses ) . In addition , we proposed a question answering system , which answers interrogative questions associated with what , by using a Web-based encyclopedia as a knowledge base . For the purpose of evaluation , we used as test inputs technical terms collected from the Class II IT engineers examination , and found that the encyclopedia generated through our method was of operational quality and quantity . We also used test questions from the Class II examination , and evaluated the Web-based encyclopedia in terms of question answering . We found that our Webbased encyclopedia improved the system coverage obtained solely with an existing dictionary . In addition , when we used both resources , the performance was further improved . Future work would include generating information associated with more complex interrogations , such as ones related to how and why , so as to enhance Webbased natural language understanding .
Existing methods that extract encyclopedic knowledge from the Web output unorganized clusters of term descriptions not necessarily related to explicit criteria while clustering is performed.
They propose to use word senses and domains for organizing extracted term descriptions on questions extracted from the Web to improve the quality.
The generated encyclopedia is applied to a Japanese question and answering system and improves over a system which solely depends on a dictionary.