There are many examples of 'wisdom of the crowd' effects in which the large number of participants imparts confidence in the collective judgment of the crowd. But how do we form an aggregated judgment when the size of the crowd is limited? Whose judgments do we include, and whose do we accord the most weight? This paper considers this problem in the context of geopolitical event forecasting, where volunteer analysts are queried to give their expertise, confidence, and predictions about the outcome of an event. We develop a forecast aggregation model that integrates topical information about a question, meta-data about a pair of forecasters, and their predictions in a deep siamese neural network that decides which forecasters' predictions are more likely to be close to the correct response. A ranking of the forecasters is induced from a tournament of pair-wise forecaster comparisons, with the ranking used to create an aggregate forecast. Preliminary results find the aggregate prediction of the best forecasters ranked by our deep siamese network model consistently beats typical aggregation techniques by Brier score.
With respect to forecast aggregation, recent work has focused on integrating forecaster features and past performance. Budescu @cite_16 proposes a method for measuring a single judge's (e.g., a single forecaster's) contribution to the crowd's performance and uses positive contributors to build a weighting model for aggregating forecasts. Forlines @cite_17 shows how heuristic rules based on the qualities of each forecaster can significantly improve aggregated predictions. Hosen @cite_10 applies a neural network to forecast a range of likely outcomes based on a weighted average aggregation of forecaster predictions. Further work by Ramos @cite_6 uses a particular nonparametric density estimation technique called L2E, which aims at making the aggregation robust to clusters of opinions and dramatic changes. The Good Judgement project by Tetlock @cite_14 introduces an approach based on a novel cognitive-debiasing training design. Atanasov @cite_26 shows that team prediction polls outperform prediction markets once forecasts are statistically aggregated using various techniques such as temporal decay, differential weighting based on past performance, and recalibration.
@cite_14 @cite_26 @cite_6 @cite_16 @cite_10 @cite_17
2108920669 2194444479 2597667148 2007423920 2136009263 2208261036
Forecasting tournaments are level-playing-field competitions that reveal which individuals, teams, or algorithms generate more accurate probability estimates on which topics. This article describes a massive geopolitical tournament that tested clashing views on the feasibility of improving judgmental accuracy and on the best methods of doing so. The tournament’s winner, the Good Judgment Project, outperformed the simple average of the crowd by (a) designing new forms of cognitive-debiasing training, (b) incentivizing rigorous thinking in teams and prediction markets, (c) skimming top talent into elite collaborative teams of “super forecasters,” and (d) fine-tuning aggregation algorithms for distilling greater wisdom from crowds. Tournaments have the potential to open closed minds and increase assertion-to-evidence ratios in polarized scientific and policy debates. We report the results of the first large-scale, long-term, experimental test between two crowd sourcing methods – prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a geopolitical prediction tournament. Some forecasters traded in a continuous double auction market and were ranked based on earnings. Others submitted probability judgments, independently or in teams, and were ranked based on Brier scores. In both seasons of the tournament, last day prices from the prediction market were more accurate than the simple mean of forecasts from prediction polls. However, team prediction polls outperformed prediction markets when poll forecasts were aggregated with algorithms using temporal decay, performance weighting and recalibration. The biggest advantage of prediction polls occurred at the start of long-duration questions. Prediction polls with proper scoring, algorithmic aggregation and teaming offer an attractive alternative to prediction markets for distilling the wisdom of crowds. Statistical aggregation is often used to combine multiple opinions within a group. Such aggregates outperform individuals, including experts, in various prediction and estimation tasks. This result is attributed to the “wisdom of crowds.” We seek to improve the quality of such aggregates by eliminating poorly performing individuals from the crowd. We propose a new measure of contribution to assess the judges' performance relative to the group and use positive contributors to build a weighting model for aggregating forecasts. In Study 1, we analyze 1,233 judges forecasting almost 200 current events to illustrate the superiority of our model over unweighted models and models weighted by measures of absolute performance. In Study 2, we replicate our findings by using economic forecasts from the European Central Bank and show how the method can be used to identify smaller crowds of the top positive contributors. We show that the model derives its power from identifying experts who consistently outperform the crowd. Data, as supplemental material, are available at http: dx.doi.org 10.1287 mnsc.2014.1909 . This paper was accepted by James Smith, decision analysis. Neural networks (NNs) are an effective tool to model nonlinear systems. However, their forecasting performance significantly drops in the presence of process uncertainties and disturbances. NN-based prediction intervals (PIs) offer an alternative solution to appropriately quantify uncertainties and disturbances associated with point forecasts. In this paper, an NN ensemble procedure is proposed to construct quality PIs. A recently developed lower–upper bound estimation method is applied to develop NN-based PIs. Then, constructed PIs from the NN ensemble members are combined using a weighted averaging mechanism. Simulated annealing and a genetic algorithm are used to optimally adjust the weights for the aggregation mechanism. The proposed method is examined for three different case studies. Simulation results reveal that the proposed method improves the average PI quality of individual NNs by 22 , 18 , and 78 for the first, second, and third case studies, respectively. The simulation study also demonstrates that a 3 –4 improvement in the quality of PIs can be achieved using the proposed method compared to the simple averaging aggregation method. The aggregate prediction from a group of forecasters often outperformsthe individual forecasts from experts. In this paper, wepresent an improvement on the traditional “Wisdom of Crowds” aggregation technique that uses extra information elicited fromforecasters along with their prediction. An analysis of 64 forecastingquestions answered by a group of 1000+ novice predictorsshows that applying heuristic weighting rules to forecasts resultsin a 28 improvement in the accuracy of the group’s predictions.