Is it true that the worse the crowd predictions the more useful the justifications? (it is).
In addition, we also present an analysis of the justifications submitted with correct and wrong forecasts to shed light into which characteristics make a justification more and less credible.
Previous Work
<FIGURE>The language people use is indicative of several attributes.
Previous work includes both predictive models (input: language samples, output: some attribute about the author)
and models that yield useful insights (input: language samples and attributes of the authors, output: differentiating language features depending on the attributes).
Among many others, previous research has studied gender and age [1]}, [2]}, [3]},
political ideology [4]}, [5]},
health outcomes [6]},
and personality traits [7]}.
In this paper, we do not profile forecasters.
Instead, we build models to call questions
based on forecasts by the crowd without knowledge of who submitted what.
Previous research has also studied the language people use to communicate depending on
the relationship between the parties.
For example, the language people use when they are in positions of power (e.g., more seniority)
has been studied in
social networks [8]},
online communities [9]},
and corporate emails [10]}.
Similarly, rashid-blanco-2018-characterizing
study how language provides clues about the interactions and relationships between people.
Regarding language form and functions,
prior research has analyzed
politeness [11]},
empathy [12]},
advice [13]},
condolences [14]}
usefulness [15]},
and
deception [16]}.
More related to the problem we work with,
maki-etal-2017-roles analyze the influence of Wikipedia editors,
and
katerenchuk2016hierarchy study influence levels in online communities.
Persuasion has also been studied from a computational perspective [17]}, [18]},
including dialogue systems [19]}.
The work presented here complements these works.
We are interested in identifying credible justifications in order to aggregate crowdsourced forecasts,
and we do so without explicitly targeting any of the above characteristics.
Within computational linguistics, the previous task that is perhaps the closest to our goal is argumentation:
a good justification for a forecast is arguably a good supporting argument.
Previous work includes identifying argument components such as
claims, premises, backings, rebuttals, and refutations [20]},
and mining supporting and opposing arguments for a claim [21]}.
Notwithstanding these works,
we found that crowdsourced justifications rarely fall into these argumentation frameworks
despite the former are useful to aggregate forecasts.
Finally, there are a few works on forecasting that use the same or very similar corpora than we do.
From a psychology perspective,
mellers2015identifying present strategies
to improve forecasting accuracy (using top forecasters, i.e., superforecasters) and analyze
the characteristics of superforecaster performance, which can be used for cultivating better forecasters.
mellers2014psychological discuss explanations of what makes forecasters better.
These works aim at identifying superforecasters and do not take into account the written justifications.
Unlike them, we build models to call questions without using any information about forecasters.
Within computational linguistics,
schwartz-etal-2017-assessing assess the
language of quality justifications (rating, benefit, and influence).
zong-etal-2020-measuring is perhaps the closest experiment to ours. They build models to predict forecaster skill using the text justifications of forecasts from Good Judgment Open data, and they also use another dataset, Company Earnings Reports, to individually predict which forecasts are more likely to be correct predictions.
Unlike us, none of these works aim at calling the question throughout its life.

Dataset
We work with data from the Good Judgment Open,https://www.gjopen.com/ a website where questions are posted
and people submit forecasts.
Questions are about geopolitics
and include topics such as domestic and international politics, the economy, and social issues.
We collected all binary questions along with all their forecasts including a prediction and a justification.
In total, the dataset we work with contains 441 questions and 96,664 forecasts submitted in 32,708 days.
This is almost twice the amount of forecasts considered by zong-etal-2020-measuring.
Since our goal is to call questions throughout their life,
we work with all forecasts with written justifications regardless
of length,
how many forecasts have been submitted by the same forecaster, etc.
Additionally, our framework preserves privacy as we do not use any information about the forecaster.
The bottom plot in Figure REF  shows the average number of daily and active forecasts over the life of all questions.
There is roughly a uniform number of forecasts submitted each day,
thus the amount of active forecasts increases linearly over the life of the question.
The majority baseline with both daily and active forecasts submitted in the previous 10 days is quite accurate,
especially towards the closing date of questions.
The experiments presented in this paper aim at calling questions throughout their life.
As we shall see, models to automatically
call questions benefit from taking into account justifications
during the first three quarters of the life of a question.
<TABLE><FIGURE>Analyzing the Questions
Table REF  shows a basic analysis of the questions in our dataset.
The majority of questions have over 16 tokens and several entities;
the most common are geopolitical, person and date entities.
Regarding the life of questions, we observe that half are open for almost two months, and 75% for over three weeks.
Figure REF  shows the LDA topics [22]} obtained with gensim [23]}.
We observe three main topics:
elections (voting, winners, candidate, etc.),
government actions (negotiations, announcements, meetings, passing (a law), etc.),
and
wars and violent crimes (groups killing, civilian (casualties), arms, etc.).
While not shown in the LDA topics,
the questions cover both domestic and international events in these topics.
<TABLE>
Analyzing the Justifications
Table REF  presents basic analysis of the 96,664 forecasts justification in our dataset.
The median length is short (1 sentence and 23 tokens),
and justifications mention named entities less often than questions (Table REF ).
We check whether justifications have negations using the cues annotated in ConanDoyle-neg [24]}.
Surprisingly, half of the justifications have one negation, and 25% have three or more.
This indicates that forecasters sometimes rely on what may not happen (or has not happened) to make predictions about the future (questions do not have negations).
We also look at the sentiment polarity of justifications using TextBlob [25]}.
The majority of justifications are neutral (polarity close to 0).
In terms of readability,
we compute the
Flesch [26]}
and
Dale-Chall [27]} scores.
Both scores indicate that around a quarter of justifications require a college education to be understood.
In terms of verbs and nouns,
we analyze them using the WordNet lexical files [28]}.
The most common verb classes are
change (26% of justifications, ,
happen, remain, increase)
social (24%, ,
vote, support, help)
cognition (22%, ,
think, believe, know)
and
motion (19%, ,
go, come, leave).
The most common noun classes are
act (71%, ,
election, support, deal),
communication (57%, ,
questions, forecast, news),
cognition (38%, ,
point, issue, possibility),
and
group (38%, ,
government, people, party).

Experiments and Results
We experiment with the problem of calling a question throughout its life.
The input to the problem is the question itself and forecasts (predictions and justifications),
and the output is an answer to the question aggregating all the forecasts.
The number of instances is the number of days all questions were open
(recall our dataset contains 441 questions and 96,664 forecasts submitted in 32,708 days).
We experiment both with simple baselines
and a neural network taking into account (a) daily forecasts
and (b) active forecasts submitted up to ten days prior.
Experimental results showed that considering earlier active forecasts is not beneficial.
We divide the questions into training, validation, and test subsets.
Then, we assign to each subset all the forecasts submitted throughout the life of the questions.
Note that randomly splitting forecasts would be unsound,
as forecasts for the same questions submitted on different days would end in the training, validation, and test subsets.
<FIGURE>Baselines
We consider two unsupervised baselines.
The majority vote baseline calls a question based on the majority prediction in the forecasts.
The weighted vote baseline calls a question after weighting the chances assigned to the predictions in the forecasts.
Consider these three forecasts:
99%, 45%, and 45% chance the answer is yes (thus 1%, 55%, and 55% chance the answer is no).
The majority vote baseline would output no (2 out of 3 believe no is more likely).
On the other hand, the weighted vote baseline would output yes
(the weighted support for yes is larger, 0.99 vs. 0.90).

Neural Network Architecture
We experiment with the neural network architecture depicted in Figure REF .
The network has three main components:
a component to obtain a representation of the question,
a component to obtain a representation of a forecast,
and an LSTM [29]} to process the sequence of forecasts and call the question.
We obtain the representation of a question using
BERT [30]}
followed by a fully connected layer with 256 neurons,
ReLU activation,
and 0.5 dropout [31]}.
We obtain the representation of a forecast concatenating three elements:
(a) a binary flag indicating whether the forecast was submitted in the day the question is being called or in the past,
(b) the prediction (a number ranging from 0.0 to 1.0),
and
(c) a representation of the justification.
We obtain the representation of the justification using BERT
followed by a fully connected layer with 256 neurons, ReLU activation, and 0.5 dropout.
The LSTM has a hidden state with dimensionality 256, and takes as its input the sequence of forecasts.
During the tuning process, we discovered that it is beneficial to pass the representation of the question with each forecast
as opposed to processing forecasts independently of the question.
Therefore, we concatenate the representation of the question to each representation of a forecast prior to feeding the sequence to the LSTM.
Finally, the last hidden state of the LSTM is connected to a fully connected layer with 1 neuron and sigmoid activation to call the question.
Architecture Ablation
We experiment with the full neural architecture as described above and disabling several components.
Specifically, we experiment with representing a forecast taking into account different information:
the prediction;

the prediction and the representation of the question;

the prediction and the representation of the justification; and

the prediction, the representation of the question, and the representation of the justification.
Implementation and Training Details
In order to implement the models,Code to replicate our experiments available at
https://github.com/saketh12/forecasting_emnlp2021
we use the Transformers library by HuggingFace [32]}
and PyTorch [33]}.
We use binary cross-entropy loss,
gradient accumulation and mixed precision training [34]}
to alleviate the memory requirements,
the Adam optimizer [35]} with learning rate 0.001,
batch size 16,
and early stopping with patience set to 3 epochs.
We tuned all the hyperparameters comparing held-out results with the validation set,
and report results with the test set.
<TABLE>
Quantitative Results
Table REF  presents the results.
The evaluation metrics is accuracy (, average percentage of days a model calls a question correctly throughout the life of the question).
We report results for all days (column 2)
and
the four quartiles (columns 3–6).
Despite their simplicity, the baselines obtain good results (71.89 and 73.79 using daily and active forecasts),
showing that aggregating the predictions submitted by forecasters without regard to the question or justifications is a competitive approach.
As we shall see, however,
the full neural network obtains statistically significant better results (79.96 and 81.27 using daily and active forecasts).
Using Daily or Active Forecasts
Taking into account active forecasts instead of only those submitted on the day the model is calling the question (daily forecasts)
is beneficial across both baselines and all neural networks except the one using only predictions + justification.
The differences in accuracy
are larger with the baselines (daily: 71.89 vs. 77.27; active: 73.79 vs. 77.97)
than with the neural networks.
We note, however, that the differences are statistically significant evaluating with all days and all quartiles except Q1 (indicated with \(^{\dagger }\)  in Table REF , McNemar's test [36]} with \(p < 0.05\) ).
We conclude that using active forecasts is beneficial
and focus the remaining of the discussion on these results.
<TABLE>
Encoding Questions and Justifications
The neural network that uses only the prediction to represent a forecast outperforms both baselines (78.81 vs. 77.27 and 77.97).
More interestingly, incorporating into the representation of the forecast
the question,
the justification,
or both brings improvements (79.35, 80.84, and 81.27).
All but the results with predictions + justifications are statistically significant with respect to using only predictions.
We conclude that calling a question benefits from incorporating into the model the question and the justifications submitted by forecasters.

Calling Questions Throughout their Life
We now move beyond accuracies calculated using all days throughout the life of a question
and examine detailed results per quartile.
More specifically,
we divide the days into four quartiles.
The last four columns in Table REF  show that while using active forecasts is beneficial across all four quartiles (with both baselines and all networks),
the neural networks—perhaps surprisingly—outperform the baselines only in the first three quartiles.
In fact, the neural networks obtain statistically significant worse results than any of the baselines in the last quartile
(84.67 vs. 87.44 and 88.22; -3.2% and -4.0%).
We conclude that modeling questions and justifications is overall useful,
although it is detrimental towards the end of the life of a question.
The justification for this empirical fact is that the crowd gets wiser towards the end of the life of a question—as more evidence to make the correct prediction presumably becomes available, and more forecasters submit forecasts.
Our model does not take into account which day is calling a question in (within the life of the question).
We reserve to future work incorporating temporal information to better aggregate forecasts.

Calling Questions Based on their Difficulty
We finish the quantitative experiments with results depending on the difficulty of the questions.
To this end,
we sort questions by their difficulty based on how many days the majority or weighted vote baselines (whichever makes the least mistakes) calls the questions wrong.
These experiments shed light into how many questions benefit from the neural networks
that take into account the question and justifications.
We note, however, that it is impossible to calculate question difficulty during the life of the question, so these experiments are not realistic before a question closes (and the correct answer is known).
After all, forecasts are about predicting the future, and it is only challenging to do so while the correct answer is unknown.
Table REF  shows the results with selected models depending on question difficulty.
We observe that the weighted vote baseline calls 75% questions more reliably than the neural network.
Indeed, the baseline obtains 99.40, 99.55, 86.01 and 29.30 accuracy in each quartile of difficulty,
while the best network obtains
95.71 (-3.7%),
93.18 (-6.4%),
79.99 (-7.0%), and
64.41 (+119.8%).
In other words,
the majority of questions (75% easiest questions) obtain worse results with the best neural network (-3.7–7.0%),
but a substantial amount (25% hardest questions) are called correctly more than twice as often (+119.8%).
The benefits with the hardest questions compensate the drawbacks with the easiest questions.
As stated earlier, overall the full neural network obtains significantly better results than the baselines (81.27 vs. 77.27 and 77.97).
We conclude that learning how to aggregate crowdsourced forecasts,
and specifically taking into account the question and justifications,
is the most beneficial with the hardest questions.

Qualitative Analysis
<TABLE>In this section, we present insights into
(a) what makes questions harder to forecast
and
(b) characteristics of justification submitted with wrong and correct predictions (Table REF ).
Questions
We looked at three characteristics of the 88 questions in the test set depending on whether the best model
(bottom row in Table REF )
calls the question at least one day wrong (it does so with 36 out of 88 questions).
Surprisingly, we found that questions that are called correct in all days
have a longer life (81.7 vs. 69.4 days) and less active forecasts per day (26.7 vs. 31.0).
As one would expect,
our best model makes mistakes with the same questions that forecasters struggle with.

Justifications.
We manually analyzed 200 justifications submitted with wrong and correct predictions (400 in total).
Specifically, we looked at predictions submitted on days that our best model makes a mistake calling the corresponding question.
Here are the observations we identified:
We found that 78% of wrong predictions were submitted with short justifications (less than 20 tokens),
while 65% of correct predictions were.
This observation corroborates that longer user-generated text has higher quality [37]}.
Example: Software isn't good enough yet,
submitted to question Will Google's AlphaGo beat world champion Lee Sedol in the five-game Go match planned for March 2016?

While relatively few forecasts refer to previous forecasts (by the same or other forecasters, or the current forecast by the crowd),
we observe that justifications for wrong predictions do almost twice as often (31.5% vs. 16.0%).
Example: Returning to initial forecast.

Lack of logical arguments is common in the justifications we work with.
This is true regardless of whether the predictions they were submitted with are wrong or correct. We found, however, that not having a logical argument is more common with wrong predictions (62.5% vs. 47.5%).
Example: I guess Greek head of state does not count, but we are getting close,
submitted to question Will Iran host a head of state or government from one of the G7 countries on an official visit before 1 July 2016?

Surprisingly, justifications with generic arguments are not a clear indicator of wrong or correct predictions (16.0% vs. 14.5%).
Example: It seems to be pretty much decided, unless something completely out of the blue happens.

Poor grammar and spelling or non-English are rare, but much more common in justification of wrong predictions (24.5% vs. 14.5%).
Example: For reference y'all and Wenn Trump den Kurs beibehAlt.
Conclusions
Forecasting is the process of predicting future events.
Government and industry alike are interested in forecasting because
it affords them the capability to anticipate and address potential challenges to come.
In this paper, we work with questions across the political, economic, and social spectrum published in the Good Judgment Open website,
and forecasts submitted by the crowd without special training.
Each forecast consists of a prediction and a justification in natural language.
We have shown that aggregating the weighted predictions of forecasters is a robust baseline to call a question through its life.
Models that take into account both the question and justifications, however,
obtain significantly better results when calling a question in the first three quartiles of its life.
Crucially, our models do not profile forecasters or use any information about who submitted which forecast.
The work presented here
opens the door to assessing the credibility of anonymous forecasts in order to come up with aggregation strategies that are robust without tracking forecasters.

Acknowledgments
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
The results presented in this paper were also obtained using the Chameleon testbed supported by the National Science Foundation [38]}.

