1. Field of the Invention
The present invention relates generally to information retrieval systems and, more particularly, to natural language information retrieval systems.
2. Related Art
As information systems become increasingly interconnected through intranets and internets, the main problem with the search for information has shifted from determining whether the requisite information exists to determining how to locate such information.
To explore the Internet or some other large database, well-known browsers and search engines are available. Unfortunately, currently existing search engines generally require the use of expressions and search methods with which the user has to be familiar, and generally require the user to enter keywords assumed to be related to the information that the user seeks. Moreover, a typical search returns a vast amount of information, much of it being irrelevant to the user. The user is then required to find the few relevant documents from the search results. Furthermore, it is not uncommon to not locate interesting documents because the user did not use keywords that corresponded to the words or word forms as written in those documents. Various conventional techniques have been developed to improve the recall of these and other information retrieval systems. However, these techniques have many drawbacks that limit their effectiveness in improving the ability of information retrieval systems to identify all available information related to a desired search topic.
A primary drawback is that the conventional method for expressing the content of a text is a single word extraction. Conventional information retrieval methods rely on word stemming or rooting, skip word filtering, and proximity measures. Typically, conventional systems stem or root the words that occur in the text and subsequently filter out all stems or roots that appear in a predetermined skip list. The skip list contains words that have little or no predictive value. Such words include function words, such as articles, pronouns, prepositions and other frequently used words. The end result of these conventional methods is a keyword list containing a list of single non-trivial words occurring in the document, optionally ordered by their frequency in the text. The keywords in the keyword list are in their stemmed or rooted form, and accompanied by their offset values or similar location markers.
For computer applications that require an intelligent representation of the content of a text, these conventional methods are inadequate. For example, meaningful units of content generally consist of more than one single word as provided by the conventional information retrieval systems. For example, in a keyword list containing, among others, "Amsterdam", "Rotterdam", "Marathon", and "Airport", the informational content of the text, the Rotterdam Marathon and Amsterdam Airport, is lost. Likewise, in a query "are the Antwerp Yellow Pages in the Web yet?", the crucial phrase evidently is "Antwerp Yellow Pages". This phrase needs to be parsed and processed in a way that retains the informational content of the query. Specifically, only those documents that literally match on "Yellow Pages" and not just any occurrence of either "page" and "yellow" separately, as well as on "Antwerp" in a premodifying or postmodifying position, rather than just in any location in the text, should be retrieved from the searched database. However, conventional information retrieval systems typically yield the query "Antwerp or Yellow or Page" which may retrieve, among others, documents on Flemisch paper factories. Furthermore, the single keywords "yellow" and "page" fail to express the notion that emerges from their combination and preservation of the plural form; that is, "Yellow Pages".
In addition, the single keyword lists used in conventional information retrieval systems do not merge expressions that are different in form but share the same reference. A method that ignores synonyms, hyponymys, name variants, frequent misspellings, and other semantic relations, fails to give a proper representation of the document's content. For example, a text dealing with the wife of the current president of the United States may contain any number of references to that person, ranging from "the President's wife" and "The First Lady" to "Mrs. Clinton" and "Hilary." Conventional information retrieval systems ignore these synonymous expressions, failing to give a proper representation of a text's contents.
Another drawback to conventional systems is that their mechanical application of a skip word list ignores the content representation of the text. For example, the individual word "page" may be a skip word, but in combinations like "Yellow Pages" or even "The Sports Pages," it should be preserved. Likewise, skipping "first" from "First Lady" leads to a loss of the essence of the expression.
What is needed, therefore, is an apparatus and method for efficiently retrieving information that accurately represents the content of both the text being searched, and the user's query, in such a way that the two can be more effectively matched.