1. Field of the Invention
This invention relates to searching for data and, more particularly, to analyzing a natural language message, determining a category in a knowledge base corresponding to the content of the message, and intelligently providing documents from the knowledge base category in response to the message.
2. Description of the Related Art
With the ubiquitous nature of the Internet, most people are familiar with searching for data using a search engine. On the Internet, a search engine is a coordinated set of programs that can include the following:                A “spider” (also called a “crawler” or a “bot”) that goes to every page or representative pages on Web sites designated as searchable and reads those pages, using hypertext links on each page to discover and read a site's other pages;        A program that creates an index (sometimes called a “catalog”) from the pages that have been read; and        A program that receives a search request, compares the search request to the entries in the index, and returns results to the user.        
The general concepts and problems encountered when searching for data are explained herein in the context of Internet searching by search engines, although the same concepts apply for searches performed on documents that are not web pages by other types of software, such as database searches by database management systems.
An alternative to using a search engine on the Internet is to explore a structured directory of topics. The Yahoo web site is a widely-used directory on the Web, although Yahoo also allows a user to use a search engine. A number of Web portal sites offer both the search engine and directory approaches to finding information.
Before a search engine can tell you where a file or document is, the file or document must be locatable. To find information on the hundreds of millions of Web pages that exist, a search engine employs spiders to build lists of the words found on Web sites. When a spider is building its lists, the process is called Web crawling. To build and maintain a useful list of words, a search engine's spider examines a very large number of web pages.
A spider typically begins with lists of heavily-used servers and very popular pages. The spider begins with a popular site, indexing the words on its pages and following every link found within the site. In this way, the spider quickly begins to travel, spreading out across the most widely used portions of the Web, to build an index.
Searching through an index involves a user building a query and submitting it through the search engine. The query can be quite simple, a single word at minimum. Building a more complex query requires the use of Boolean operators that allow the user to refine and extend the terms of the search.
The searches defined by Boolean operators, such as AND, OR, NOT, and so on, are literal searches—the search engine looks for the words or phrases exactly as they are entered. This characteristic can be a problem when the entered words have multiple meanings. “Bed,” for example, can be a place to sleep, a place where flowers are planted, the storage space of a truck, or a place where fish lay their eggs. If the user is interested in only one of these meanings, the user might not want to see pages featuring all of the others. The user can build a literal search that tries to eliminate unwanted meanings, but it is preferable that the search engine assist with the elimination.
One of the areas of search engine research is concept-based searching. Some of this research involves using statistical analysis on pages containing the words or phrases in the search, in order to find other pages in which the user might be interested. The information stored about each page is greater for a concept-based search engine than for a keyword searching engine, and far more processing is required for each search. Still, many groups are working to improve both results and performance of this type of search engine. Others have moved on to another area of research, called natural language queries.
The idea behind natural language queries is that the user can type a question in the same way the user would ask the question to a human being; there is no need to keep track of Boolean operators or complex query structures. A popular natural language query web site today is AskJeeves.com, which parses the query for keywords that it then applies to the index of sites it has built. AskJeeves.com only works with simple queries; but competition is heavy to develop a natural-language query engine that can accept a query of great complexity.
All three of these types of search engines search for words in the index matching the words used in the search query. As mentioned above, even concept-based searching typically involves analysis of actual words in the result document and matches those word combinations to words in the search text. The analysis and searching in a context-based search is typically much slower and more expensive than keyword searching.
What is needed is a way to combine the advantages and capabilities of these different searching techniques. Preferably, the user should be able to enter his or her query in natural language text, rather than as a Boolean expression. The search engine should provide intelligence to search based on concepts, rather than solely based on words appearing in both the search query and the search result. In addition, the search engine should find documents quickly and present the results in order of perceived relevance to the search query.