Combining the system's predictions with the crowd using a weighted average--4x weight for the crowd, which we find optimal on the validation set--improves the overall Brier score from.149 to.146 on the full test set (Table 3, top row).