Q: what debates dataset was used?
Text: Introduction
Public debates are a common platform for presenting and juxtaposing diverging viewpoints As opposed to monologues where speakers are limited to expressing their own beliefs, debates allow for participants to interactively attack their opponents' points while defending their own. The resulting flow of ideas is a key feature of this conversation genre.
In this work we introduce a computational framework for characterizing debates in terms of conversational flow. This framework captures two main debating strategies—promoting one's own points and attacking the opponents' points—and tracks their relative usage throughout the debate. By applying this methodology to a setting where debate winners are known, we show that conversational flow patterns are predictive of which debater is more likely to persuade an audience.
Case study: Oxford-style debates. Oxford-style debates provide a setting that is particularly convenient for studying the effects of conversational flow. In this competitive debate format, two teams argue for or against a preset motion in order to persuade a live audience to take their position. The audience votes before and after the debate, and the winning team is the one that sways more of the audience towards its view. This setup allows us to focus on the effects of conversational flow since it disentangles them from the audience's prior leaning.
The debate format involves an opening statement from the two sides, which presents an overview of their arguments before the discussion begins. This allows us to easily identify talking points held by the participants prior to the interaction, and consider them separately from points introduced spontaneously to serve the discussion.
This work is taking steps towards better modeling of conversational dynamics, by: (i) introducing a debate dataset with rich metadata (Section SECREF2 ), (ii) proposing a framework for tracking the flow of ideas (Section SECREF3 ), and (iii) showing its effectiveness in a predictive setting (Section SECREF4 ).
Debate Dataset: Intelligence Squared
In this study we use transcripts and results of Oxford-style debates from the public debate series “Intelligence Squared Debates” (IQ2 for short). These debates are recorded live, and contain motions covering a diversity of topics ranging from foreign policy issues to the benefits of organic food. Each debate consists of two opposing teams—one for the motion and one against—of two or three experts in the topic of the particular motion, along with a moderator. Each debate follows the Oxford-style format and consists of three rounds. In the introduction, each debater is given 7 minutes to lay out their main points. During the discussion, debaters take questions from the moderator and audience, and respond to attacks from the other team. This round lasts around 30 minutes and is highly interactive; teams frequently engage in direct conversation with each other. Finally, in the conclusion, each debater is given 2 minutes to make final remarks.
Our dataset consists of the transcripts of all debates held by IQ2 in the US from September 2006 up to September 2015; in total, there are 108 debates. Each debate is quite extensive: on average, 12801 words are uttered in 117 turns by members of either side per debate.
Winning side labels. We follow IQ2's criteria for deciding who wins a debate, as follows. Before the debate, the live audience votes on whether they are for, against, or undecided on the motion. A second round of voting occurs after the debate. A side wins the debate if the difference between the percentage of votes they receive post- and pre-debate (the “delta”) is greater than that of the other side's. Often the debates are quite tight: for 30% of the debates, the difference between the winning and losing sides' deltas is less than 10%.
Audience feedback. We check that the voting results are meaningful by verifying that audience reactions to the debaters are related to debate outcome. Using laughter and applause received by each side in each round as markers of positive reactions, we note that differences in audience reception of the two sides emerge over the course of the debate. While both sides get similar levels of reaction during the introduction, winning teams tend to receive more laughter during the discussion ( INLINEFORM0 ) and more applause during the conclusion ( INLINEFORM2 ).
Example debate. We will use a debate over the motion “Millennials don't stand a chance” (henceforth Millennials) as a running example. The For side won the debate with a delta of 20% of the votes, compared to the Against side which only gained 5%.
Modeling Idea Flow
Promoting one's own points and addressing the opponent's points are two primary debating strategies. Here we introduce a methodology to identify these strategies, and use it to investigate their usage and effect on a debate's outcome.
Identifying talking points . We first focus on ideas which form the basis of a side's stance on the motion. We identify such talking points by considering words whose frequency of usage differs significantly between the two teams during the introduction, before any interaction takes place. To find these words, we use the method introduced by monroe2008fightin in the context of U.S. Senate speeches. In particular, we estimate the divergence between the two sides' word-usage in the introduction, where word-usage is modeled as multinomial distributions smoothed with a uniform Dirichlet prior, and divergence is given by log-odds ratio. The most discriminating words are those with the highest and lowest z-scores of divergence estimates. For a side INLINEFORM0 , we define the set of talking points INLINEFORM1 to be the INLINEFORM2 words with the highest or lowest INLINEFORM3 -scores. We distinguish between INLINEFORM5 's own talking points INLINEFORM6 , and the opposing talking points INLINEFORM7 belonging to its opponent INLINEFORM8 . These are examples of talking points for the “Millennials” debate:
The flow of talking points . A side can either promote its own talking points , address its opponent's points, or steer away from these initially salient ideas altogether. We quantify the use of these strategies by comparing the airtime debaters devote to talking points . For a side INLINEFORM0 , let the self-coverage INLINEFORM1 be the fraction of content words uttered by INLINEFORM2 in round INLINEFORM3 that are among their own talking points INLINEFORM4 ; and the opponent-coverage INLINEFORM5 be the fraction of its content words covering opposing talking points INLINEFORM6 .
Not surprisingly, we find that self-coverage dominates during the discussion ( INLINEFORM0 , INLINEFORM1 ). However, this does not mean debaters are simply giving monologues and ignoring each other: the effect of the interaction is reflected in a sharp drop in self-coverage and a rise in opponent-coverage once the discussion round begins. Respectively, INLINEFORM2 and INLINEFORM3 , both INLINEFORM4 . Examples of self- and opponent-coverage of two talking point s in the “Millennials” debate from the introduction and discussion are given in Table TABREF9 .
Does the change in focus translate to any strategic advantages? Figure FIGREF11 suggests this is the case: the drop in self-coverage is slightly larger for the side that eventually wins the debate ( INLINEFORM0 ). The drop in the sum of self- and opponent-coverage is also larger for winning teams, suggesting that they are more likely to steer away from discussing any talking points from either side ( INLINEFORM1 ).
Identifying discussion points. Having seen that debaters can benefit by shifting away from talking points that were salient during the introduction, we now examine the ideas that spontaneously arise to serve the discussion. We model such discussion points as words introduced to the debate during the discussion by a debater and adopted by his opponents at least twice. This allows us to focus on words that become relevant to the conversation; only 3% of all newly introduced words qualify, amounting to about 10 discussion points per debate.
The flow of discussion points . The adoption of discussion points plays an important role in persuading the audience: during the discussion, eventual winners adopt more discussion points introduced by their opponents than eventual losers ( INLINEFORM0 ). Two possible strategic interpretations emerge. From a topic control angle BIBREF0 , perhaps losers are more successful at imposing their discussion points to gain control of the discussion. This view appears counterintuitive given work linking topic control to influence in other settings BIBREF1 , BIBREF2 .
An alternative interpretation could be that winners are more active than losers in contesting their opponents' points, a strategy that might play out favorably to the audience. A post-hoc manual examination supports this interpretation: 78% of the valid discussion points are picked up by the opposing side in order to be challenged; this strategy is exemplified in Table TABREF14 . Overall, these observations tying the flow of discussion points to the debate's outcome suggest that winners are more successful at using the interaction to engage with their opponents' ideas.
Predictive Power
We evaluate the predictive power of our flow features in a binary classification setting: predict whether the For or Against side wins the debate. This is a challenging task even for humans, thus the dramatic reveal at the end of each IQ2 debate that partly explains the popularity of the show. Our goal here is limited to understanding which of the flow features that we developed carry predictive power.
Conversation flow features. We use all conversational features discussed above. For each side INLINEFORM0 we include INLINEFORM1 , INLINEFORM2 , and their sum. We also use the drop in self-coverage given by subtracting corresponding values for INLINEFORM3 , and the number of discussion points adopted by each side. We call these the Flow features.
Baseline features. To discard the possibility that our results are simply explained by debater verbosity, we use the number of words uttered and number of turns taken by each side (length) as baselines. We also compare to a unigram baseline (BOW).
Audience features. We use the counts of applause and laughter received by each side (described in Section SECREF2 ) as rough indicators of how well the audience can foresee a debate's outcome.
Prediction accuracy is evaluated using a leave-one-out (LOO) approach. We use logistic regression; model parameters for each LOO train-test split are selected via 3-fold cross-validation on the training set. To find particularly predictive flow features, we also try using univariate feature selection on the flow features before the model is fitted in each split; we refer to this setting as Flow*.
We find that conversation flow features obtain the best accuracy among all listed feature types (Flow: 63%; Flow*: 65%), performing significantly higher than a 50% random baseline (binomial test INLINEFORM0 ), and comparable to audience features (60%). In contrast, the length and BOW baselines do not perform better than chance. We note that Flow features perform competitively despite being the only ones that do not factor in the concluding round.
The features selected most often in the Flow* task are: the number of discussion points adopted (with positive regression coefficients), the recall of talking points during the discussion round (negative coefficients), and the drop in usage of own talking points from introduction to discussion (positive coefficients). The relative importance of these features, which focus on the interaction between teams, suggests that audiences tend to favor debating strategies which emphasize the discussion.
Further Related Work
Previous work on conversational structure has proposed approaches to model dialogue acts BIBREF3 , BIBREF4 , BIBREF5 or disentangle interleaved conversations BIBREF6 , BIBREF7 . Other research has considered the problem of detecting conversation-level traits such as the presence of disagreements BIBREF8 , BIBREF9 or the likelihood of relation dissolution BIBREF10 . At the participant level, several studies present approaches to identify ideological stances BIBREF11 , BIBREF12 , using features based on participant interactions BIBREF13 , BIBREF14 , or extracting words and reasons characterizing a stance BIBREF15 , BIBREF16 , BIBREF17 . In our setting, both the stances and the turn structure of a debate are known, allowing us to instead focus on the debate's outcome.
Existing research on argumentation strategies has largely focused on exploiting the structure of monologic arguments BIBREF18 , like those of persuasive essays BIBREF19 , BIBREF20 . In addition, tan+etal:16a has examined the effectiveness of arguments in the context of a forum where people invite others to challenge their opinions. We complement this line of work by looking at the relative persuasiveness of participants in extended conversations as they exchange arguments over multiple turns.
Previous studies of influence in extended conversations have largely dealt with the political domain, examining moderated but relatively unstructured settings such as talk shows or presidential debates, and suggesting features like topic control BIBREF0 , linguistic style matching BIBREF21 and turn-taking BIBREF22 . With persuasion in mind, our work extends these studies to explore a new dynamic, the flow of ideas between speakers, in a highly structured setting that controls for confounding factors.
Limitations and Future Work
This study opens several avenues for future research. One could explore more complex representations of talking points and discussion points , for instance using topic models or word embeddings. Furthermore, augmenting the flow of content in a conversation with the speakers' linguistic choices could better capture their intentions. In addition, it would be interesting to study the interplay between our conversational flow features and relatively monologic features that consider the argumentative and rhetorical traits of each side separately. More explicitly comparing and contrasting monologic and interactive dynamics could lead to better models of conversations. Such approaches could also help clarify some of the intuitions about conversations explored in this work, particularly that engaging in dialogue carries different strategic implications from self-promotion.
Our focus in this paper is on capturing and understanding conversational flow. We hence make some simplifying assumptions that could be refined in future work. For instance, by using a basic unigram-based definition of discussion points , we do not account for the context or semantic sense in which these points occur. In particular, our annotators found that a significant proportion of the discussion points under our definition actually referred to differing ideas in the various contexts in which they appeared. We expect that improving our retrieval model will also improve the robustness of our idea flow analysis. A better model of discussion points could also provide more insight into the role of these points in persuading the audience.
While Oxford-style debates are a particularly convenient setting for studying the effects of conversational flow, our dataset is limited in terms of size. It would be worthwhile to examine the flow features we developed in the context of settings with richer incentives beyond persuading an audience, such as in the semi-cooperative environment of Wikipedia talk pages. Finally, our methodology could point to applications in areas such as education and cooperative work, where it is key to establish the link between conversation features and an interlocutor's ability to convey their point BIBREF23 .
Acknowledgements. We thank the reviewers and V. Niculae for their helpful comments, and I. Arawjo and D. Sedra for annotations. This work was supported in part by a Google Faculty Research Award.
“Intelligence Squared Debates” (IQ2 for short)