Figure 4: **System strengths. Evaluating on the validation set, we note: (a) When provided enough relevant articles, our system outperforms the crowd. (b) For questions where the crowd is unsure (predictions between \(.3\) and \(.7\)), we outperform them (Brier score \(.199\) vs. \(.246\)). However, the crowd outperforms our system on questions where they are highly confident, e.g. predicting less than \(.05\). (c) Our system’s Brier score is better at the earlier retrieval dates.