Data stored on one or more computer systems may be useful for a user. However, the stored data may be too extensive for a user to find the data by direct examination. Additionally, some parts of a data repository may contain information that is not accessible to the user. In many cases, in order to allow the user useful access to the data, a search mechanism is provided. The search mechanism allows a user to issue a search request (also termed a search query). The query is executed and the results are returned for the user.
For example, a web-based search engine is a search mechanism that is used to provide search access to information via a web-based search. The information may be found in a specific data repository, such as a database or other data collection. The information may also be an agglomeration of data found in a number of different data repositories. Such a search engine may provide search access to information available from different information providers over a network, such as the Internet.
In a typical usage of a web search engine, the user enters a query, which is a set of search terms related to what the user is looking for. The query is transmitted to the search engine, which attempts to locate “hits”—i.e., content that is available on the Internet and that relates to the terms contained in the query. Generally, the search engine either has a database of web pages that are known to exist, or communicates with external “providers” who maintain such databases; the query is “scored” against items in these databases to identify the web pages that best match the query. A list of results is then generated, and these results are returned to the user's computer for display by the user's web browser.
Typically, the databases contain information such as: the Uniform Resource Locators (URLs) of web pages, the titles of the pages, descriptions of the pages, and possibly other textual information about the web pages. The user then reads the results and attempts to determine, based on the text contained in the results, whether the results correspond to what the user is looking for. Users may then attempt to retrieve the entire page correlating to a search result. In other contexts, search engines present results summarizing the pieces of data that may possibly be useful for a user.
The utility of the search engine is correlated directly to the quality of the results provided. In the best case, the results are presented to the user in order of utility to the user on the result page. Because the quality of the results is subjective, the user's satisfaction must be determined in order to determine whether the quality of the results were satisfactory.
In the prior art, quality of individual web pages has been measured by obtaining explicit feedback from a user. At least one prior art web browser has attempted to obtain such explicit feedback from a user. This browser is described in a paper entitled “Inferring User Interest” by Mark Claypool, David Brown, Phong Le, Makoto Waseda in IEEE Internet Computing 5(6): 32-39 (2001). In that browser, different pages are displayed by the browser. Whenever the page being displayed by the browser is changed, a user evaluation of the page is requested from the user. User evaluations for a given page are collected, to determine whether users find that page valuable. In this browser, some implicit feedback is also maintained regarding each page, including data regarding the time spent on the page, mouse movements, mouse clicks, and scrolling time.
While this technique does gather user feedback, it has limited utility in situations in which users may have different needs for a page. For example, a user who is looking for information about books written by Douglas Adams may evaluate a page on his book The Hitchhiker's Guide to the Galaxy and give a high score for utility. However, another user who is looking for information on books about traveling cheaply may evaluate the same page and give it a low score. Thus the technique described will have limited utility in the wide variety of situations in which different users may have different needs, or even where a single user may have different needs for information at different times. In other words, the usefulness of this technique is limited because evaluation of each page is completely independent of the context in which the user arrived at the page.
Thus, this technique is not useful for evaluating the quality of a search engine. In general, this technique is not useful for evaluations that are context-based, but only for evaluating the quality of individual data items, independent of the context in which a user arrived at the data items.
Another drawback of the prior art is that users may respond to explicit requests for feedback by ignoring them or providing inaccurate feedback in order to speed the searches the users are performing. This leads to possibly unreliable data in addition to user dissatisfaction with the search tool.
Additionally, the prior art web browsers do not request feedback on non-standard search results. Many web searches now provide non-standard search results. For example, when searching web pages, for example, for a restaurant of a specified name in a specified town, some web searches simultaneously do a phone book search for the phone number of the restaurant. This result is displayed for the user. This is a non-standard result because it is not the type of result that the user was ostensibly requesting. However, it may be useful to judge the satisfaction of the user with such non-standard results.
An additional problem in judging satisfaction with search results is that they may be used by a user without the user selecting the result (standard or non-standard) and without the user otherwise indicating through other implicit feedback that the user has found a satisfactory result. In the above example, a user who is only looking for a web page regarding the restaurant to find the phone number of the restaurant may find that phone number in a non-standard result and use that information without selecting any result, standard or non-standard.
Another drawback of the prior art is that feedback from each user is considered without reference to the user making the feedback. A user with a different experience level, language, purpose, or technical ability may have different responses to the search mechanism, however there is no way to include such differences in considering user satisfaction according to the prior art systems.
Thus, there is a need for a system and method to overcome these deficits in the prior art. The present invention addresses the aforementioned needs and solves them with additional advantages as expressed herein.