Visual Abductive Reasoning

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, Reasoner (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative representations for the premise and hypothesis. Then, multiple decoders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR benchmarking results show that Reasoner surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.


Introduction
Abduction · · · consists in studying facts and devising a theory to explain them.
- Charles Sanders Peirce (1839-1914 Abductive reasoning [51] was coined by Charles Sanders Peirce, the founder of American pragmatism, around 1865. It is inference to the most likely explanation or conclusion for an incomplete set of observations. Abductive reasoning is invariably employed in our everyday life; the generated hypothesis (H) is expected to best explain what happens before, after, or during the observation (O). Fig. 1 gives some examples. If you see O: "the road is wet", abduction * Part of this work was done when Chen Liang was an intern at Baidu. will lead you to the best explanation H: "it rained earlier" (i.e., H →O). One morning you find O: "sister leaves home hurriedly", then you conclude H: "she will be late for class" (i.e., O → H). You see O 1 : "a boy throws a frisbee out and his dog is running after it". One minute later you find O 2 : "frisbee is in the boy's hand". You can imagine H: "the dog just caught the frisbee back" (i.e., O 1 → H → O 2 ). Although abductive reasoning has long been considered as a core ability of everyday human cognition [40,57,59], it still remains an untouched domain in computer vision literature.
In this article, we propose Visual Abductive Reasoning (VAR), a novel task and dataset for investigating the abductive reasoning ability of AI systems in daily visual situations. In particular, inspired by the recent advance of causal reasoning in NLP community (i.e., abductive text generation [5] and counterfactual story revision [52]), we explore the use of natural language as the expression form to fully capture the complexity of real situations. This also better reflects the nature of human mind, which relies on linguistic thinking [38,39]. VAR requires the AI systems to describe the incomplete observation (i.e., visual premise) and write down the hypothesis that can best explain the premise. This allows to thoroughly evaluate the entire abduction procedure, as accurate understanding of the premise is the basis of abductive reasoning. Moreover, this can hasten the development of this new field, by comparing and embracing ideas for a relevant, well-established, yet different task -dense video captioning (DVC) [30]. In contrast to DVC that focuses only on describing the observation, VAR yields a new visual-linguistic reasoning paradigm -inference beyond observation. Three characteristics make VAR uniquely challenging: i) VAR needs imagination to find hypotheses outside the observation. ii) VAR seeks to discover the plausible causal structure among the observed events. iii) VAR is more related to the kind of human reasoning in daily situations where the information at hand is often incomplete [26] and absolutely certain conclusions cannot be reached [5,27].
Our dataset is collected to address the characteristics of the VAR task (cf. §3). It contains 9K examples from 3,718 videos. Each example consists of several chronologicallyordered events, most of which are logically related. For each event, abduction oriented description is written by people, and its role of premise or explanation is also annotated. When presenting each example to the AI system, the explanation event is masked and premise events are visible. The AI system is required to understand the partial, noisy observations (i.e., premise events) and construct the most plausible explanatory hypothesis -accurately describing both the observable premise events and the masked explanation event. The examples are on average 37.8s long, with 4.17 events, and harvested from diversely event-rich sources, i.e., YouTube Lifestyle videos, movies and TV shows.
To lay a solid foundation for future efforts, a new model, named REASONER (causal-and-cascaded reasoning Transformer), is proposed (cf. §4). Specifically, REASONER is building upon a Transformer encoder-decoder architecture. In the encoder of REASONER, a contextualized directional position embedding strategy is adopted to capture causal dependencies among the premise events. Hence the context of the premise events can be gathered in a causality-aware manner, enabling REASONER to learn discriminative representations for the premise and explanatory hypothesis. Then REASONER cascades a set of decoders for premise/hypothesis sentence production and refinement. For each generated sentence, the associated prediction score is viewed as the confidence and embedded into the next decoder as a signal for inspiring more information to flow from highconfident sentences to the low-confident ones. This leads to a confidence-guided multi-step reasoning strategy, boosting the reasoning power of REASONER eventually.
Extensive experimental results are provided in §5. First, to comprehensively probe deep neural models on this challenging task, we establish a group of baselines based on current top-leading DVC models. The benchmarking results on VAR dataset show that REASONER outperforms the best competitor by a large margin, e.g., 33.44 vs 28.71 in terms of BERT-S, but is still far behind human performance (42.96). This shows that VAR is especially challenging for current video-language models. Then a set of user studies and ablative experiments are conducted for a thorough evaluation. For completeness, we further test REASONER on the DVC task and confirm again its superiority. Concurrent to us, [17] studies image-based abductive reasoning: AI systems are required to identify, ground, or compare given inferences. Overall, we feel vision-based abductive reasoning is an intriguing problem worthy of exploring.

Related Work
Dense Video Captioning (DVC). Different from the classic video description task [29,46,67,68,76,80] that aims to describe a short video clip using a single sentence, DVC is to comprehensively describe all the events in an untrimmed video through a multi-sentence paragraph [30]. Typical DVC models [30,[43][44][45]60,74,81,82] follow a two-stage, bottomup paradigm: first parse a video into several temporal events and then decode a description from each detected event. As the problem of event detection is ill-defined [10], some alterative solutions either adopt a single-stage strategy to simultaneously predict events and descriptions [35,71], or turn to a top-down regime: first generate paragraphs, and then ground each description to a video segment [10,37]. A few other methods [23,32,50] focus purely on generating better paragraph captions from a provided list of events.
Both VAR and DVC are concerned with video-based text generation; a part of our dataset is sourced from Ac-tivityNet Captions [30], a famous DVC dataset. However, DVC is aware of general fact based plain narrative, while VAR addresses cause-effect chain based abductive reasoning. Rather than accurately understanding what is observed, VAR further requires invoking what might have happened or will happen. In our experiments, we involve several recent DVC models as baselines for our VAR task and also report the performance of our REASONER on the DVC task. Context-Aware Text Generation. Our work is also related to some context-aware text generation tasks in the NLP literature. For instance, text infilling [83], also known as the cloze task [64], is to generate a span of missing tokens in a text chunk, while sentence/story infilling [19,21] aims to generate missing sentences in long-form text. The generated tokens/sentences are expected to smoothly blend into and fit the context syntactically [83], semantically [19,21], and logically [24]. Counterfactual story revision [52] requires generating a new ending, given a story context altered by a counterfactual condition. Our work draws inspiration from abductive text generation [5], which investigates abductive reasoning via a natural language inference task: write an appropriate reason that could explain observations described by narrative text. Unlike these language tasks addressing inter-sentential relationship understanding only, our VAR task requires abduction and narrative for a sequence of partially observable visual events. Moreover, our VAR task setting is more general; it is not limited to the strict form of abductive reasoning in [5] Visual Future/State Prediction. Our work is, to some degree, relevant to future prediction -a popular research area in computer vision. In this area, a huge spectrum of topics/tasks are put forward, including forecasting future frames [42,70], future features [62,69], future actions [1,28,31,55,61], future human motions [14,22,41], future human trajectories [2], future goals [13], etc. Rather than studying the future generation at the semantic-category or color-pixel level, event-level prediction was recently addressed in [34] and [49]. However, [34] only requires choosing from two candidates for future event prediction, making the take less challenging. [49] targets to describe past, present, and future events for a single image, while our VAR task requires making full use of the information from a set of premise events. There are also some efforts made towards understanding the dynamics/transformations between states [18,48,75] or goal-conditioned procedure planning [7], while either relying on a pre-defined, limited action prediction space [7,18], or using simulated environments [7,18]. Our VAR task is essentially to discover and describe the causal relations in real visual environments. It is not constrained to a narrow view of predicting either future events or between-state changes, but tries to infer the missing parts in the cause-effect chains, even with some unrelated premise events. All of these together make VAR a unique and challenging visual reasoning task. Position Encoding in Transformers. Due to the permutation invariant nature of the attention operation, [58] learns and encodes position embeddings into Transformer tokens. Subsequent language-Transformers hence explore further variations, like incorporating sinusoid prior with more parameters [9], simplifying position embeddings as learnable scalars [53], disentangling special tokens ([CLS]) [25], etc. Some recent vision-Transformers [8,73] consider directional relative distance between 2D positions, and/or the interactions between visual tokens and position embeddings. REASONER encodes the relations of premise events in a directional and contextualized manner for causal relation modeling, and leverages the prediction scores of descriptions for confidence-guided multi-step reasoning.

VAR Task
Our visual abductive reasoning (VAR) task is designed to test the abductive reasoning ability of machine intelligence in everyday visual environments E. Formally, given a video example V ⊂ E that contains a set of N events, i.e., V = {O 1 , · · ·, O n−1 , H, O n , · · ·, O N −1 }, which are logically related and chronologically organized, we denote {O n } N −1 n=1 as premise events -partial observation of E, and H as explanation event that can best explain the premise events. Conditioning on the past and/or future visual con-  text {O n } N −1 n=1 only, the AI system is asked to describe these premise events, and, more importantly, infer and write down the most plausible explanatory hypothesis for the premise. Naturally, such a hypothesis is expected to be consistent with the content of the explanation event H. The abduction ability can thus be thoroughly examined by assessing the quality of both the premise-based descriptions and explanatory hypothesis sentences -adequent understanding of the premise is a necessary prerequisite for abductive reasoning.

VAR Dataset
Guided by the above task setup, we build a large-scale dataset for VAR. Fig. 2 depicts an illustrative example.

Dataset Collection
Data Source. VAR dataset is collected from three sources: • 23,457 Youtube lifestyle Vlog videos from ActivityNet Captions [30] and VLEP [34] datasets. These videos cover rich social scenarios and human activities. • 13,799 TV show and movie videos from TVC dataset [33] and a famous Youtube channel, Fandango MovieClips 1 . These videos are key scenes in popular TV shows and films containing wide-ranging genres. YouTube videos include diverse daily events, but last relatively short durations with short intervals (about minutes). While TV shows and movie videos usually have limited scenarios, they contain rich artificial cause-effect chains in their story-lines and last relatively long durations with long intervals (spanning even years). Thus gathering these videos together makes our dataset a good testbed for VAR. Data Cleaning. The collected videos are accompanied by event labels, and videos containing only one single event are first dropped. Then, for each of the rest videos, three human experts are invited to examine if there exists cause-effect relations between the video events. We only preserve qualified ones with more than two votes in the affirmative, finally resulting in 3,718 videos in total for further annotation.

Dataset Annotation
For each video in VAR, the annotation contains three steps: Step 1: Event Type Annotation. For an event E of video V, if E can well explain some other events in V; or in other words, if we can imagine that E could happen by only considering the other events V/E, event E will be labeled as  explanation and the other events V/E will be labeled as premise. Fig. 2 gives an example. For the video containing three events: E 1 "a man falls off a running horse", E 2 "the man lies on the ground and gets hurt", E 3 "the man is taken to the hospital", we can derive two legal examples for our VAR dataset: Step 2: Abductive Reasoning Aware Description Annotation. Although some videos are collected with event-level descriptions/plot summaries, we re-annotate all the events with abductive reasoning oriented descriptions. Specifically, instead of capturing all the visual details in video captioning, like {"a boy in a black jacket plays frisbee with his white dog in a park", "the dog catches the blue frisbee and runs back", "the boy smiles, takes the frisbee and pats the dog}, our descriptions are only aware of describing the visual content related to abductive reasoning, like {"a boy throws a frisbee out and his dog is running after it", "the dog catches the frisbee back", "the boy gets the frisbee"}.
Step 3: Validation. Finally, each annotated example is examined by three verifiers: the verifiers are shown with only the premise events and language-based explanation (i.e., description on the explanation event), and vote for: "Is the explanation sound?". If an example wins majority approval, it will be accepted; otherwise, it will be relabeled or dropped.

Dataset Features and Statistics
To offer deeper insights into our VAR dataset, we next discuss its distinctive properties and detailed statistics. Abductive Reasoning Orientated. VAR is the first dataset that underpins machine intelligence study of abductive reasoning in visual daily scenarios. It is designed to reason beyond visual premise for a plausible explanation, distinguishing it from existing video-language datasets/tasks. Diversity. To capture diverse cause-effect relations and abduction cases, our VAR dataset covers i) various daily events/activities, e.g., work, leisure, household; ii) rich scenarios, e.g., lifestyle recording, scripted drama; iii) different durations and intervals, ranging from minutes to years. Large-Scale. As shown in Table 1

Methodology
Problem Statement. Given a video V with N temporally ordered events, i.e., , and explanation event, i.e., H, are logically related. The AI system is only presented with a partially observable version of V, i.e.,Ṽ = {O 1 , · · ·, O n−1 ,H, O n , · · ·, O N −1 }, whereH is obtained by setting all the pixel values of H as zero. The AI system is required to not only describe the premise, but also reason about the most likely explanation for the premise, i.e., generate N sentences S={S O n } N −1 n=1 ∪S H that describe the content of the N events in V, while conditioning onṼ only: where w l is the l-th word in a generated sentence S ∈ S. It is worth mentioning that, when H = ∅, our VAR task is degraded into a classic DVC task [30] which focuses only on describing the content of observed events {O n } N −1 n=1 . Core Idea. Building upon a Transformer encoder-decoder architecture (Fig. 4), our REASONER is aware of two core challenges posed by the VAR task: i) inferring cause-effect relations, and ii) reasoning beyond the partial observation. To address i), a contextualized directional position embedding strategy is adopted to capture causal relations residing in the input videoṼ, leading to a Causality-aware encoder ( §4.1). To accommodate ii), a confidence-guided multistep reasoning strategy is developed, i.e., utilize the prediction scores of sentences to guide cross-sentence information flow, yielding a cascaded-reasoning decoder ( §4.2).

Causality-Aware Encoder
For notational simplicity, we redefine the partially observable videoṼ={O 1 , where E h refers to the masked explanation eventH, and {E n } =h indicates the visible, premise events {O n } N −1 n=1 . Let us denote the initial features of the N events as For each premise event E n =h , corresponding feature E n =h is obtained by aggregating the visual features of its frames. For the masked explanation event E h , we set E h = 0 d . The Causality-aware encoder is to leverage the context from the past and/or future observable events {E n } =h to reinforce their own representations as well as posit a meaningful representation for the most likely explanatory hypothesis, i.e., the masked explanation event E h .
The attention operation is the core of Transformer: where the output Y ∈ R N ×d is with the same length N and embedding dimension d as the input X ∈ R N ×d , and  Figure 4. Network architecture of REASONER. See §4 for more details. W q,k,v ∈ R d×d project the input into query, key, and value matrices, respectively. As the attention computation is invariant with respect to reordering of the inputs, explicit position encoding is widely adopted, in two typical ways: i) Absolute position encoding [65]: each position n is assigned an embedding, i.e., U n =F Abs (n) ∈ R 1×d , and the position embeddings are directly added to the input, i.e., X ← X+U . F Abs (·) can be a linear projection [12], a sinusoidal function [65], etc. ii) Relative position encoding [58]: the position embeddings are constructed considering the pairwise relationships between positions, i.e., U nm = F Rel (n, m) ∈ R. Contextualized Directional Position Embedding. Since the VAR task is essentially aware of the plausible chains of cause-effect, the relative ordering of the input events matters. We continue in the vein of relative position encoding [58,73] and adopt a contextualized directional position embedding strategy, i.e., U nm = F Rel (n, m, X n ) ∈ R: where R ∈ R (2N −1)×d is a learnable matrix, and (·, ·) is a directional indexing function, i.e., (n, m) = (m, n). The directional projection F Rel is conditioned on the visual context, i.e., X n , since the causal dependency between events is typically related to specific content, e.g., when we see people are laughing, we tend to look back only a short time into the past to figure out the reason; when we see a man falls off his horse, we worry about whether he gets hurt and the impact on his future life. Some more visual examples regarding our contextualized directional position embedding strategy can be found in Fig. 5. Then, U∈R N×N is injected by manipulating on the attention matrix A∈R N×N : We further set A nh = 0 to encourage leveraging the context from the observable events {E n } =h to infer the masked explanation event E h , rather than vice versa. The Causalityaware encoder in REASONER is therefore achieved by stacking several Transformer encoder blocks [65] with our contextualized directional position embedding strategy. We denote the output event representations as {Ṽ n ∈ R d } N n=1 .

Cascaded-Reasoning Decoder
With the discriminative representations {Ṽ n } N n=1 of the observable premise events {O n } N −1 n=1 as well as the explanatory hypothesisH, the cascaded-reasoning decoder first generates a descriptive sentence for each event/hypothesis individually, and then refines all the sentences in a comprehensive, confidence-guided, and step-by-step manner. Initial Description Generation. For each event representa-tionṼ n ∈ R d , a multi-modal, masked Transformer decoder is first adopted for initial description generation: where H n ∈R Ln×d is a set of L n words embeddings. During training, it is computed over the groundtruth description, i.e.,Ŝ En , and masked attention [65] is adopted to prevent the leakage of future words. During inference, it is recurrently generated. Learnable modal-type embeddings [11,32] are also added into the input yet omitted for brevity. By fusing visual and linguistic representations as the input, D 0 conducts cross-modal reasoning, and hence generates improved event representation, i.e.,Ṽ 0 n ∈ R d , and updated visual-linguistic state, i.e., H 0 n ∈ R Ln×d , for each event E n . Then a captioning head is adopted to map H 0 n into word distribution. The probability of l-th word is given as: where Ω ∈ R |Ω|×d is the embedding matrix of the word vocabulary Ω, and H 0 n (l)∈R d denotes l-th vector of H 0 n . As standard, the description S 0,En = {w En l } Ln l=1 for event E n is generated by greedy prediction, and we set the averaged prediction score as the confidence: c 0 n = 1 Ln i P (w En l ). Iterative Description Refinement. To better respond to the fundamental challenge of VAR task in reasoning beyond observation, we further cascade several Transformer decoder blocks over D 0 for iterative description refinement. This allows REASONER to make full use of both visual and linguistic context from the past and/or future observable events, and improves the explanatory hypothesis in a stepby-step manner, boosting the reasoning ability eventually.
Sentence from Sentence from time : He is using a brush to polish the wood surface.
: He is using a brush to cover each picket in a stain.
: He then wipes the table with a towel.
: The picket is stained and is waiting to dry. Figure 6. Sentences from the cascaded-reasoning decoder ( §4.2).
Specifically, our whole refinement procedure can be defined in a recursive, confidence-guided form: where D k refers to k-th refinement module and all the refinement modules are weight-sharing Transformer de- In this way, each D k can leverage inter-sentential relationship in previously generated descriptions {h k−1 n } N n=1 for refinement and better reason about the explanatory hypothesis. Moreover, we introduce the event confidence, i.e., {c k n } N n=1 , as a kind of bias into the refinement procedure: leverage the information from those more confident descriptions to help improve the predictions with relatively lower confidence. Without causing ambiguity, we denote X as the input of the decoder and omit the superscript k. For each input "token" X i , its confidence score c ni is the one of its sourced event E ni , and we normalize {c n } N n=1 over all the N events. Analogous to Eq. 4, the attention computation in D k is modified as: where the learnable vector r c ∈ R 2B−1 can be viewed as a bucket to store the relative confidence weight; and the directional indexing function ι(·, ·) is given as ι(c ni , c nj ) = c ni ·B − c nj ·B +B. With such confidence-guided decoding scheme, descriptions are refined by intelligently gathering context from more reliable sentences, while ignoring noisy cues from less confident ones. By stacking several such decoders {D k } k , outputs will be progressively improved (Fig.6). Related experiments can be found in §5.2.

Training Objective
Given the groundtruth sentences {Ŝ En } N n=1 corresponding to the N events {E n } N n=1 of videoṼ, REASONER is trained by minimizing the negative log-likelihood over the outputs of the cascaded-reasoning decoder {D k } K k=0 : Ln l=1 P (ŵ En l |ŵ En <l , H k n ), (9) whereŜ En ={ŵ En l } Ln l=1 . As the teacher forcing scheme [72] is used for training, H n in Eq. 5 and 7 is embedded over one-hot encoded groundturth words {ŵ En l } l . We further adopt a hypothesis reconstruction based optimization criterion, to provide the encoder with more explicit supervision signals for explanatory hypothesis reasoning: whereṼ h andV h are embeddings for the explanatory hypothesis obtained from the masked and original videos, i.e., V and V, respectively, and F Proj is a projection head, based on a small multi-layer perceptron. This auxiliary training objective forces REASONER to "imagine" an effective rep-resentationṼ h that better aligns with the original content of E h .V h is from the momentum version of the encoder.

Implementation Details
Details on implementing the algorithm are as follows: • Detailed network architecture: The encoder ( §4.1) of REASONER is implemented as two Transformer encoder blocks, and each decoder module ( §4.2), i.e., D k , is implemented as two Transformer masked decoder blocks. They have d = 768 hidden size and 12 attention heads. We use a bucket size B = 10 to quantize confidence scores (Eq. 8).
We stack a total of K = 3 decoders for cascaded reasoning.
• Data preprocessing: For each video event, action/appearance features are pre-extracted using ActivityNet [6] pretrained ResNet200 [15]/BN-Inception [20], as in [32,71,82]. We uniformly sample 50 frames per event and concatenate their features as the corresponding event representation which is denoted in a vector form in §4.1-4.3 for ease of notation. Sentences are padded or truncated into 20 words.
• Training/Inference: For the first decoder D 0 , we adopt scheduled sampling [4] to make the later decoders fully trained. The coefficient between the main and auxiliary training objectives is set as 0.2. During inference, the final descriptive sentences are generated from the last decoder D K , using deterministic decoding, i.e., greedy search. All the experiments are conducted on 2 NVIDIA GeForce RTX 2080 Ti GPUs with a 11GB memory per-card.

Experiments
We first provide benchmarking results on our VAR dataset ( §5.1). Then, to verify the efficacy of our core model designs, we conduct a set of diagnostic studies ( §5.2). Finally, for comprehensive evaluation, we test our REASONER on the classic, dense video captioning (DVC) task [30] ( §5.3).

Performance on VAR Task
Competitor. We benchmark five top-leading DVC models on VAR to reveal the abductive reasoning ability in existing techniques. They include three Transformer-based [9,32,82] and two RNN-based [71,74] models, which are trained on train set of our VAR dataset with pre-provided event segments using their original training protocols.  Table 3. User study of pairwise model preference (%). "Neutral" means A and B models are "equally good". Percentage of "equally bad" are omitted. See §5.1 for details.
Quantitative Result. Table 2 summarizes the benchmarking results on the test set of our VAR dataset. For detailed analysis, we report the performance over the observable premise events and invisible explanation events separately. Moreover, to probe the upper bound of model performance, we evaluate human performance by asking ten volunteers to perform VAR. Specifically, we randomly sample 500 examples from unique videos in VAR test. The volunteers are only provided with partially observable videos and requested to write down the corresponding descriptions and hypotheses. The human-written descriptions and hypotheses are evaluated by the automatic metrics, and evaluation scores are shown in the first row of Table 2. Several essential conclusions can be drawn from Table 2: i) Humans are good at VAR; although human-written hypotheses for explanation scored lower than the descriptions for the visual premise, they are still very plausible in absolute terms. ii) All traditional DVC models [9,32,71,74,82] struggle with VAR that humans excel at. Their generated hypotheses are usually untrusted, and far worse than their created premise narratives. This suggests that existing video-based language generation models are not good at reasoning beyond observation. iii) Our REASONER outperforms other AI models [9,32,71,74,82], in both explanatory hypothesis reasoning and premise description, demonstrating the effectiveness of our whole model design. Compared to other AI models, REASONER also yields a relatively smaller performance drop, from premise description to hypothesis reasoning. This suggests that REASONER can make better use of the context of observed events to infer the explanatory hy-pothesis. iv) Although our REASONER shows more promising results, there still remains a significant gap from human performance, that is waiting for more sophisticated abductive reasoning models to conquer.
User Study. For comprehensive performance assessment, we further carry out a subjective evaluation, based on pairwise model comparison. Specifically, we randomly sample 500 examples from unique videos in VAR test. Three volunteers are presented the outputs of a pair of systems (i.e., REASONER vs PDVC [71] or human) on the sampled examples, and requested to do a comparison about which one is better, or "equally good" or "equally bad". The human preference results are collected in Table 3, and again the statistics for premise events and explanation events are presented separately. The human subjective judgments are generally accordant with the trends reflected by Table 2. Specifically, the human pairwise comparison results confirm the superiority of REASONER over PDVC, the second-best model in Table 2: REASONER receives 34.2 and 29.9 percent preference votes on the premise description and explanatory hypothesis, respectively. However, human-written hypotheses and descriptions are much more favorable than our results, showing again VAR is a very challenging task.
Qualitative Analysis. A test video example in VAR dataset is shown in Fig. 13. It contains the explanatory hypotheses and premise descriptions from our REASONER and other competitors [32,71,82] as well as groundtruth sentences. We can find that our REASONER is able to discover and correctly describe the cause-effect chain, and hence generate a plausible hypothesis, i.e., making a small splash, that well explains the observed events, i.e., standing on the podium. In contrast, other competitors typically produce unsatisfactory results, especially for the explanatory hypothesis.

Diagnostic Experiment
A set of ablative studies is conducted on VAR test for indepth analyzing each component in our REASONER, using BLEU@4, CIDEr and BERT-S metrics, averaged over all the events. Key Component Analysis. We first study the efficacy of core model components. The first row in Table 8a gives the performance of a basic Transformer model, which simply uses ab- solute position embedding in the encoder and only adopts one single decoder, i.e., D 0 . The results in the first two rows reveal that contextualized directional position embedding ( §4.1) consistently improves the performance over the three metrics. Moreover, from the first and third rows we can observe that confidence-guided multi-step reasoning ( §4.2) indeed boosts the performance. By further considering the scores in the last row, we can safely conclude that combining the two model designs together leads to the best results. Contextualized Directional Position Embedding. Next, to thoroughly study the impact of our contextualized directional position embedding strategy ( §4.1), we report the performance of two alternatives in Table 8b. Specifically, "absolute" refers to the widely used, learnable absolute position embedding, while "directional" indicates learning relative position embedding without considering any input context. As seen, our contextualized directional position embedding is significantly better than the two alternatives. Cascaded Reasoning. Table 8c reports the performance with different steps of our cascaded reasoning ( §4.2), i.e., K = {0, 1, · · ·, 5}. When K = 0, only one decoder D 0 is adopted and the CIDEr score is just 32.72. However, after adding an extra refinement decoder, the score is greatly improved to 36.13. The increasing trend is gradually saturated until K > 3. We therefore use K = 3 as our default setting for balancing performance and inference efficiency. Confidence Embedding. We inject sentence scores into the cascaded reasoning for guiding information flow (Eq. 8). As shown in Table 8d, removing confidence embedding hinders the performance, e.g., 36.13→35.22 in terms of CIDEr. Training Objective. Finally we examine our training objective design ( §4.3). Table 8e demonstrates a beneficial impact of the hypothesis reconstruction loss L Aux (Eq.10).

Performance on DVC Task
For completeness, we report performance on DVC task. Dataset. As a gold-standard dataset for DVC, ActivityNet Captions [30] Table 5. Quantitative results ( §5.3) on the ae-val set of Activi-tyNet Captions [30]. The scores are mainly borrowed from [71].
(10,009/4,917/5,044 for train/val/test). Each video lasts 120s and is annotated with 3.65 temporally-localized sentences on average. Following [32,60,82], val set is further split into two subsets: ae-val with 2,460 videos and ae-test with 2,457 videos without overlapping. Evaluation Metric. As in [32,60,82], BLEU@4 [47], ME-TEOR [3], and CIDEr [66] metrics are used for evaluation. Quantitative Result. REASONER is trained on the train set and evaluated on ae-val set in paragraph-level. Since we focus only on descriptive quality, the sentences are generated from a provided list of events, like in [23,32,50]. As shown in Table 5, REASONER outperforms state-of-the-art DVC models over all the metrics, e.g., +2.81 performance gain in CIDEr. This proves the strong reasoning ability of REASONER and emphasizes the value of our VAR task in promoting innovations of powerful video-language models.

Conclusion
We introduce VAR (Visual Abductive Reasoning) -a novel task that investigates the abductive reasoning ability of machine intelligence in the visual world. We establish REASONER, a new Transformer based visual-language model, which captures the context from visual premise in a causality-aware manner, and generates premise descriptions and hypothesis sentences in a confidence-guided, step-bystep fashion. REASONER shows promising results on both VAR and dense video captioning tasks. We also observe a remaining large headroom for AI systems in VAR, which is expected to encourage exciting avenues in the future.
In this work, we build a headway of Visual Abductive Reasoning (VAR) as a new task and introduce a large-scale dataset for VAR that scaffolds the investigation of Abductive Reasoning ability in AI systems. Along with them, REA-SONER is further presented as a modest solution. In the supplemental material, we provide the following items that shed deeper insight on the aforementioned contributions:

A.1. Detailed Dataset Statistics
Our VAR dataset is curated from three main sources, i.e., YouTube Lifestyle video, movie and TV show, in Fig. 8b and Fig. 8c, we illustrate the detailed distribution of videos and examples by collected sources. As seen, most videos in VAR are from YouTube Lifestyle video, while movie videos tend to have more events, and thus leads to more complicated causal structures and more examples. We then study the distribution of frequently used words in VAR descriptions, which is illustrated as a word cloud in Fig. 8a. More frequent words are shown in larger font size. Finally, the distribution of premise events is shown in Fig. 8d.

A.2. Detailed Annotation Process
In the annotation process, each video is passed at least four times: (1) A first quick pass for filtering out videos without cause-effect relations; (2) The second pass for annotating the event type, i.e., the premise and explanation. In this phase, the entire video with initialized events and descriptions are all shown to the experts. We use the original event annotation provided in [30,33,34] to initialize event boundaries, while they might contain noise annotations. We thus request human experts to i) edit event boundaries when the initial separation can not well fit the description; ii) delete duplicate events when they are overlapped; iii) add additional events when they find a missing part in the cause-effect chain. And the annotation interface is shown in Fig. 9. After that, experts are requested to further annotate the event type based on the events he selected in the previous step, as shown in Fig. 11. (3) The third pass for abductive reasoning oriented description annotation. Human experts are only shown with premise visual events while the hypothesis event is hidden. Annotation interfaces for annotating the premise and hypothesis are shown in Fig. 10 and Fig. 12 respectively. Experts might vote to delete an example if they find that a plausible explanation can not be inferred from the premise. And the remaining examples are re-annotated with abductive reasoning oriented descriptions. Notably, a video might contain multiple examples (candidate cause-effect chains), and we manually control the example distribution to make sure the same video will not be shown to the same expert twice. (4) Forth, the final pass for validation. Both the annotated descriptions for premises and explanations for hypotheses are shown to another group of human experts. And they will vote for the validity.

B. Additional Details of Baselines
We benchmark five top-leading DVC models [9,32,71,74,82] on VAR. In this section, we detail the implementation and training protocol of these baseline methods.  MFT MFT [74] is an LSTM-based method that consists of a selection LSTM for relevant event filtering and a captioning LSTM for coherent sentence generation. We adapt it to VAR task by recurrently passing the given events into MFT and the selection LSTM is transformed into a visually coherent maintaining module. The unidirectionally casual structure can be captured that enables the inference on potential effect along the temporal order.
PDVC Similarly, PDVC [71] employs an LSTM-based captioning decoder, while it is conditioned on a deformable soft attention aggregated visual event. And the visual events are embedded with a Transformer-based encoder that could also capture bidirectional causal dependencies within it.  VTrans VTrans [82] is a fully attentional model that originates from the vanilla Transformer proposed in [65]. We follow the implementation in [32], which serves as a baseline that only considers a single event and independently generates a single sentence describing the given event. Thus causal structure can not be formulated in this method.
Trans-XL Transformer-XL (Trans-XL) [9] is originally proposed for modeling unlimited longer-term dependencies with a segment-level recurrent strategy. It can capture the intrinsic unidirectional causal structure within recurrent steps. Following the implementation in [32], gradients can flow through recurrent steps instead of being stopped. This enables stronger long-term modeling. MART MART [32] is also built on a fully Transformerbased encoder-decoder architecture, that maintains a summarized memory module to model dependencies among events. Similar to Trans-XL, the unidirectional causal structure is preserved in the memory. Whereas, experimental results show that it suffers more on our VAR potentially due to the content drift brought by masked visual hypotheses.
MFT and PDVC are benchmarked following the original training protocols. We adapt VTrans, Trans-XL and MART to the VAR task with the implementation provided by [32]. All of these baselines are trained with given events from our VAR train and evaluated on VAR test under the same setting of REASONER as reported in our main paper.

C. Additional Experimental Results
To shed light on the essence of both the Visual Abductive Reasoning task and our VAR dataset, we study two edge cases of the main setting: i) First, all events are made available to the model, so that no abductive reasoning is needed. And the VAR task is degraded to a basic Dense Video Captioning task. ii) Second, only ground-truth linguistic descriptions of premise events are supplied to the models. Therefore, models are expected to conduct abductive reasoning with and only with linguistic cues.
In Table 6, we summarize the quantitative results of these two settings. As seen, when there is no hidden event, i.e., the incomplete causal structure is directly provided, REA-SONER achieves even better performance. It reveals that fulfilling explanation events through abductive reasoning is indeed challenging and the causal structure understanding is also helpful to basic visual recognition. And for the next setting, when only premise texts are given, comparing to fully utilize both visual and linguistic cues, REASONER can not well-infer the hypothesis within linguistic modality only, which proves that the visual-based abductive reasoning is indispensable in the VAR task.
In our main paper, for the benchmarking results, we report the average scores of ten trained models with different random seeds. To prove the statistical significance of our results, here we further provide the corresponding standard deviations of REASONER and two representative methods [32,71] in Table 7.  Table 7. Average scores and their standard deviations of REA-SONER and two representative methods [32,71] ( §C).

D. Additional Qualitative Visualization
In Fig. 13-14, we show more qualitative examples from VAR test following Fig. 7 in the main paper. Generated sentences from competitors [32,71,82] along with our REA-SONER are presented in Fig. 13. In contrast to the competitors, both adequate descriptions on premises and plausible inferences for hypotheses are observed for our proposed method, REASONER, which demonstrates a superior abductive reasoning ability for capturing causal structures among visual events. Some failure cases and gold human-written explanations are shown in Fig. 14. Even though REASONER shows impressive performance on inferring with abduction, VAR task is still a mostly unsolved technical problem. And there remains a large headroom for future works to conquer.

E.1. Dataset Limitation
During annotation process, we observe a bias against women and minorities due to the highly biased nature of movie and web sourced video data [54,56,63]. VAR, derived from these data, inevitably runs into the same problem [16,77]. We thus suggest that models trained on VAR dataset should be cautiously examined before being deployed onto real-world applications. And we will devote further efforts to mitigating the issue in our later works.

E.2. Details of BERTScore Evaluation
BERTScore [79] leverages the pre-trained contextual embeddings from BERT-based models for similarity measurement. Thus the evaluated scores vary a lot with different model settings. In this paper, all reported BERTScores are evaluated under a hash code version: roberta-large L17 noidf version=0.3.0(hug trans=2.3.0)-rescaled. We encourage later works to follow the same setting for a fair comparison. A static version of BERTScore is released at: https://github.com/leonnnop/VAR.

F.2. Concerns on Personal Data Collection
VAR is annotated by human experts and we conduct user studies to evaluate the human-subjective generation quality. All human experts are noticed that the annotation and evaluation will be used for academic research and individual consents are reached with signed agreements. The annotation will not leak any personal information about the experts.

F.3. Potential Societal Impact
Endowing an AI system with human intelligence has long been dreamed by AI researchers, which could fundamentally change the experience of human-machine interaction. VAR takes an important step towards more human-like AI systems that are endowed with abductive reasoning ability, while it might provoke concerns about disinformation, e.g., fabricating deceptive facts. We encourage more technical researching efforts devoted to fake content detection, and at the same time, we will organize a gated release of our dataset and model to prevent potentially malicious abuses.