There is great interest in supporting imprecise queries over databases today. To support such queries, the system is typically required to disambiguate parts of the user-specified query against the database, using whatever resources are intrinsically available to it (the database schema, value distributions, natural language models etc). Often, systems will also have a user-interaction log available, which can supplement their model based on their own intrinsic resources. This leads to a problem of how best to combine the system's prior ranking with insight derived from the user-interaction log. Statistical inference techniques such as maximum likelihood or Bayesian updates from a subjective prior turn out not to apply in a straightforward way due to possible noise from user search behavior and to encoding biases endemic to the system's models. In this paper, we address such learning problems in interactive data retrieval, with specific focus on type classification for user-specified query terms. We develop a novel Bayesian smoothing algorithm, Bsmooth, which is simple, fast, flexible and accurate. We analytically establish some desirable properties and show, through experiments against an independent benchmark, that the addition of such a learning layer performs much better than standard methods.
. Many models combine forecasts from several human experts as independent information sources on uncertain events. They give their probability estimates, which would be compatible with relative frequencies obtained from multinomial counts like in a user interaction log. A popular approach to combine the individual forecasts is the so-called linear opinion pooling.' It assigns each forecast a weight that reflects the importance or quality of that expert. Recently, along these lines, more advanced techniques have been developed @cite_3 . One important point of improvement is to consider sharpness,' which rewards how close to either 0 or 1 an estimate is (ibid.), somehow in line with our choice to use entropy as a measure. Yet their core measure of source quality is still calibration' (see, e.g., Brier scores). http: en.wikipedia.org wiki Brier_score . Suppose an expert estimates 0.3 as the probability for a specific outcome, then she is best calibrated if that outcome is seen in 30 is dependent on seen examples.
@cite_3
2091855602
This paper begins by presenting a simple model of the way in which experts estimate probabilities. The model is then used to construct a likelihood-based aggregation formula for combining multiple probability forecasts. The resulting aggregator has a simple analytical form that depends on a single, easily-interpretable parameter. This makes it computationally simple, attractive for further development, and robust against overfitting. Based on a large-scale dataset in which over 1300 experts tried to predict 69 geopolitical events, our aggregator is found to be superior to several widely-used aggregation algorithms.