Computer Science
Computer Science-2021.gz:143135
Computer Science
7f649b926c89b466dfb4e683975c6e644a375846
Logically at Factify 2022: Multimodal Fact Verification

This paper describes our participant system for the multi-modal fact verification (Factify) challenge at AAAI 2022. Despite the recent advance in text based verification techniques and large pre-trained multimodal models cross vision and language, very limited work has been done in applying multimodal techniques to automate fact checking process, particularly considering the increasing prevalence of claims and fake news about images and videos on social media. In our work, the challenge is treated as multimodal entailment task and framed as multi-class classification. Two baseline approaches are proposed and explored including an ensemble model (combining two uni-modal models) and a multi-modal attention network (modeling the interaction between image and text pair from claim and evidence document). We conduct several experiments investigating and benchmarking different SoTA pre-trained transformers and vision models in this work. Our best model is ranked first in leaderboard which obtains a weighted average F-measure of 0.77 on both validation and test set. Exploratory analysis of dataset is also carried out on the Factify data set and uncovers salient patterns and issues (e.g., word overlapping, visual entailment correlation, source bias) that motivates our hypothesis. Finally, we highlight challenges of the task and multimodal dataset for future research.


Introduction
Rapidly growing volume of misinformation and fake news have become a pressing challenge and cause severe consequences on society. Significant joint efforts have been undertaken by a wide range of parties (represented by journalists, researchers, independent fact checkers) to protect communities from false information. It has never been more important to have a versatile ecosystem to scale up and speed up fact checking against misinformation using technology, which can be broadly categorised into claim detection and claim validation [1]. The former technique is to support fact checkers in content prioritisation through assessing check-worthiness, and the latter one is to automate the process of evidence retrieval from large knowledge bases and performing veracity prediction of the detected claims in order to assist manual fact checking tasks. Claim matching [2] is another emerging trend, addressing the need for timely identification of previously fact-checked claims. Prior efforts focused mostly on text from news media articles and English language. In recent years, with the advance in user-generated content and increasingly polarized social platforms, the challenges of fact checking have increasingly become multilingual and multimodal which have been pervasive in user-generated multimedia content [3]. As a consequence, many new problems arise, typically false context, false connections or misleading content [4,5,6,7,8]. Another understudied process (known as amplification) is leveraged by coordinated disinformation campaigns [2]. It deliberately spreads large volumes of repeated claims in many different ways, in order to stimulate unintentional spread as false rumors [8]. Thus, there is an imperative need to develop algorithms to group same claims resided in various multimodal context and automate the verification process at scale.
Compared to text-based fact checking, multimodal verification is an under-explored area of research. Image and text both contain rich information but reside in heterogeneous modalities. Comparing representation learning within the same modality, cross-modal architectures need not only learn the features for image and text to express their respective content but importantly capture a measure for cross-modal semantic integrity [8,9]. We study multimodal entailment in this paper. As a newly introduced subtask, it poses extra critical challenges.
Simple image similarity cannot resolve fine-grained images differences and perform poorly for adversarial images [10], SAR images [11], etc. To exemplify this challenge, two pairs of claim and document from insufficient multimodal samples in Factify dataset [12] are presented in Table 1. The first sample shows two separate images of a politician taken in direct point-of-view, sitting at the exact same table, in the exact same room, giving a televised speech on different days for different issues. In both images, the politician is wearing a suit, in one image black, and in the other white. In this case, the images are likely to yield high similarity with respect to their content, but they should be considered different images and representative of different contextual information. The second sample presents two images of the same nature, where the politician is wearing the same white suit and ear plugs but with a news broadcasting logo overlaid on the upper right corner of claim image, and the other document image with no news channel logo visible. The main discrepancy is presented between text and image in document which reports that the politician is wearing a white mask during the video conference. Therefore, although the document text provide supporting evidence about the claim but the image is missing important context information. On the contrary, the sample in Table 2 presents two images having low content overlap but the document image corresponds to its textual content that supports the politician death information as presented in claim image. Thus, the right image should be considered as supporting image and representative of same information contextually with corresponded claim image.
Relying on visual similarity analysis alone for multimodal fact verification is naturally prone to false positives, because images related to branding and advertisements (e.g., the "breaking news" image or a company's logo) are often reused. This may cause erroneous detection when there is no real connection between them other than the reuse of a generic image. The problem becomes more complex with images exploited in disinformation on social media.
Our work in this competition is in response to current online misinformation multimodal  issue and has focused on solving the above challenges. Two different algorithms are designed for the task that is framed as a multimodal entailment prediction problem following two different frameworks, including an ensemble learning and an end-to-end attention network. The ensemble model approach is implemented with a decision tree classifier that combines predictions of two uni-modal models with a few data-specific heuristic features. Two uni-modal models are implemented including a 3-way text entailment model based on a State-of-the-Art (SoTA) pre-trained transformer language architectures fine-tuned on the task dataset, and a pre-trained CNN model (ResNet-50) for image similarity. A SoTA multimodal attention network for 5-way end-to-end entailment classification is implemented as an alternative solution in attempt to infer combinatorial entailment relation by combining representation of language and vision. Globe-level multimodal interactions are modeled with a popular multi-branch attention network framework in order to fuse multimodal information. Strong baselines are implemented for both 3-way and 5-way text entailment models to prove the advantage of our proposed methods. Exploratory data analysis and bias test experiments are conducted to understand the potential data issues and present the challenge of creating high-quality multimodal datasets for the real-world problem. Best results from the ensemble model were submitted for the competition.
In the remainder of this paper, we firstly present a brief overview of related work (Section 2), then task definition and our proposed methods in details (Section 3) followed by experiments on the task dataset (Section 5). Exploratory data analysis is elaborated in Section 4. Finally, the results discussion (including 3-way models and 5-way models) and conclusion are provided in Sections 6 and 7 respectively.

Related Work
Text Entailment Recognising Textual Entailment (RTE) is earliest and most related work to our Factify challenge that aims to determine an inferential relationship between natural language hypothesis and premise. On the basis of a given sentence pair, the task is to predict 3-way labels including Support, Refute or NotEnoughInfo. Well-known shared tasks include FEVER [13] and SCIVER [14], which advanced RTE research for claim validation in recent years. This line of work performs different forms of evidence retrieval and then applies claim validation based on that evidence. In contract, evidence retrieval is not required in the Factify task (although the practice of sentence retrieval [15,16] as a classic NLI problem for long document text in Factify data can be considered as good practice and applicable). Stance detection is another direction of work supported by shared tasks such as UKP Snopes [17], Semeval-2017 Rumoureval [18]), and has also been exploited for RTE by retrieving texts relevant to a claim or story, determining the stance of those text afterwards so as to ultimately predict the veracity of a given claim. The common practice of RTE for claim verification [1] is also incorporated in our ensemble model (as one of the proposed solutions) and treated as a three-way text classification task on text data. Sentence retrieval for evidence aggregation and stance detection are not exploited in this work.
Multi/cross-modal representation learning In the field of multimodal reasoning and matching, the success of attention mechanism in the NLP community motivated computer vision techniques to shift from traditional twin network (typically with Siamese nets [19,20,21]) to pre-train models in multimodal settings for wide range of downstream tasks, such as visual question answering (VQA), visual reasoning and image captioning. Similar to BERT [22], the recent approach is to use a single transformer architecture to jointly encode text and image such as VisualBERT [23], Uniter [24] and VL-BERT [25]. Alternatively, ViLBERT [26] and LXMERT [27] introduced the two-stream architecture, where two transformers are applied to images and text independently, which is fused by a third transformer in a later stage. These models typically rely on region-based image features extracted by pre-trained object detectors based on commonly used two-staged detectors (typically Faster R-CNN model [28] or its extension Mask-RCNN [29]), or single-stage detectors (typically SSD and YOLO V3 [30]) or anchor-free detectors(e.g., [31]). Another directions are patch embedding [32,33,34,35,36]. This direction of work directly operates on patches (as a sequence of tokens with fixed length). Image patches and text token embeddings are fed into a transformer or self-attention model to learn fused crossmodal attention. The great progress of these recently developed models can be witnessed on the leader boards of various tasks without using ensembling such as VQA, GAQ [37], NLVR2 [38], which can mainly be attributed to the availability of large scale weakly correlated multimodal data (typically captioned images or video clips and accompanying subtitles [39]) that can be utilised to learn cross-modal representation by contrastive learning [40]. However, existing pretrained models use mostly scene-limited image-text pairs with short and relatively simple descriptive captions for images, while ignoring richer uni-modal text data and domain-specific information. This leads to the difficulties in comprehending long paragraphs than short text [41]. Thus, most such task (e.g., VQA, VAC, image retrieval) still need additional fusion layer to model interaction between visual and linguistic content. Moreover, limited ground truth information forces many tasks to use evaluation metrics based on binary relevance. Different from most of current cross-modal reasoning tasks, our work aims to model long sequence text and images for claim verification. In contrast to these multimodal architectures, we utilize the individual components from uni-modal pre-trained architectures. The equivalent architecture is employed by [42] for image-text pair interaction, however, we exploited richer cross-modal interaction among vision and text pairs. Inspired by the practice in [43], a stacked attention mechanism is exploited in our solution for cross-modal matching by inferring the latent language-vision alignments at a global level. Recent advance in fine-grained cross-modal representation learning approaches for region-word correspondence are not exploited in this work.
Relevance matching technique Relevance matching (RM) is the core problem of information retrieval (IR) and has also been applied for the detecting the entailment relation [44,45] by computing the best alignment of hypothesis to premise based on local and global interactions. Vo and Lee [46] exploited a neural ranking model using textual and visual modalities to match multimodal claim with fact-checked information. Their model unifies textual and visual interaction between a claim and a collection of candidate articles, while Factify task aims to match a claim with one given candidate document. In our proposed solution, we extended the matching module introduced in Vo and Lee [46] in order to better handle text with vary length.
Visual Entailment Visual Entailment (VE) [47] is a variant of traditional RTE task that consists of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence. The problem that VE is trying to solve is to reason about the relationship between an image as premise P and a text as hypothesis H . This is different from Factify task that aims to reason about the multimodal relationship between a hypothesis and premise pair of both textual and visual content with respected to five categories. Moreover, the premise text is of vary lengths rather than short hypothesis sentences in SNLI-VE dataset.

Problem Statement
We frame the Factify task as a problem of multimodal entailment, which is to reason about the relationship of a multimodal claim as hypothesis and a multimodal document as premise.
Specifically, given a multimodal hypothesis (e.g. tweet) denoted by = + and a document (typically one or more fact-checking articles) denoted by = + , both of which contain one image and a text, we aim to derive a function ( , ) that infers their entailment of five categories ("Support_Multimodal", "Support_Text", "Insufficient_Multimodal", "Insufficient_Text", "Refute"). The label assignment is based on the relationship conveyed by ( , ), • Support_Multimodal holds if there is enough evidence in to conclude that is true and is relevant to and in the same information context, • Support_Text holds if there is enough evidence in to conclude that is true but is irrelevant to and does not provide supplemental information for , • Insufficient_Multimodal holds if the evidence in is insufficient to draw a conclusion about but is relevant to and in the same information context, • Insufficient_Text holds if the evidence in is insufficient to draw a conclusion about and is irrelevant to and does not provide supplemental information for , • otherwise, the relationship is Refute, implying that there is enough evidence in to conclude that is false and is irrelevant to Q of both visual and text content.
Additional details of the task definition can be referred in [12].

3-way Text Entailment
Recognizing entailment in natural language is a straightforward application for fact verification.
In this section, we aim to study how well a SoTA textual entailment model can be fine-tuned on the textual data pairs in Factify data set and then used in a three-way RTE task. This allows us to further assess and benchmark our proposed solution of combining two uni-modal models prediction into an ensemble model for final 5-way multimodal entailment prediction. Pretrained transformer fine-tuning: Pretrained transformer models [22,48,49] have become the de facto models for a wide range of NLP tasks and provide SoTA results for RTE tasks [50]. More specifically, in this work, we attempt to investigate how a pretrained model can learn to conduct RTE on the given dataset without utilizing hidden dataset bias, how efficiently it can learn and generalise on the test set. The problem is different from existing benchmark datasets (MultiNLI [51], SNLI [52], Adversarial-NLI [53]) mostly consisting of short sentences. Fact verification task requires to apply natural language inference (NLI) on long paragraphs or articles. As mentioned above, to simplify the problem, the practice of evidence sentence selection [54] that are commonly adopted in SoTA evidence-aware fact checking system are not included in our study. Thus, supported maximum sequence length and optimum document context size are two of key factors to be considered.
Transformer-based models, such as BERT, have been one of the most successful deep learning models for NLP, but one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. For that reason, Google's BigBird model is selected in this study, which is one of the most successful long-sequence transformers that supports sequence length of 4000 tokens. To deal with the limitations that other models face, BigBird uses a sparse attention mechanism that reduces the quadratic dependency to linear [55]. That means that it can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, these models drastically improve performance on various NLP tasks such as claim verification for long sequences [56]. The model is fine-tuned as pair-wise classification task on re-purposed data samples converted from 5-way categories to three-way categories. Formally, given a pair of text sequence (denoted as and ) from and , we aim to fine-tune a pre-trained model to map any pair of ( , ) to a label , that determines the pre-defined textual entailment relationship ("support", "refute", "insufficient") between and . The problem is treated as a supervised learning task and a set of training examples in the form of ( , , ) is given. [ ] is added as a separator between the two inputs in pre-processing and a softmax classifier is added on top of the [ ] token in the last layer to make predictions.
MatchPyramid: In contrary to computationally expensive transformer models, we propose a simple baseline text entailment model with relevance matching technique. Intuitively, an article may be relevant to a claim if they have overlapping words or similar words. A strong interaction model, known as MatchPyramid [57,58,59,44], is adopted in our baseline model. This technique leverages a similarity matrix plotted from the similarity between a pair of sequences and a CNN with pooling strategies to extract hierarchical interaction patterns. CNNs strength of modeling spatial (position-aware) correlation is utilised for vary length among data. This deep neural network enables us to find the matching patterns between a piece of short text in claim and a long document, which is critical to problems in our task. Multiple layers of 2D convolutions and pooling are used followed by a feed-forward network. [59] experimented with four similarity functions (indicator function, dot product, cosine and gaussian kernel) and found that using embedding, gaussian kernel similarity function is better than others. A proper kernel size will get more information and generate a better result. Pooling size are used to reduce the dimension of the feature maps, and to pick out the most important information for the latter layers. Especially in ac-hoc retrieval task, documents often contain hundreds of words, but most of them might be background words (exactly same problem in our task). So the pooling layers might be even more important to distill the useful information from the noisy background.
Inspired by [47,46], and are pre-processed and embedded with pre-trained word embedding model. The embeddings are used to initialise the network. Self-attention layer is applied to embeddings of two inputs since the premise document ( ) in Factify can be very long and complex. Intuitively, self-attention can be helpful to capture structural information and focus on important keywords particularly in long distance dependencies. Specifically, the scaled dot product (SDP) attention [60] is used to capture this hidden information: where here represents or , denotes the embedding dimension, ∈ R × is the claim text ( ) feature matrix and ∈ R × is the document text ( ) feature matrix. and is sequence length (of embedding) in matrix and .
∈ R × computes resulting attention mask for and respectively.
Subsequently, the self-attended and feature matrix is then fed into a GRU layer in order to obtain contextual representation. Finally, dot product function is applied to build similarity matrix between two GRU output sequences for MatchPyramid model in attempt to measure the semantic relevance more accurately between claim and document with higher level of word semantics. The output of MatchPyramid is flattened into a 1D vector and fed into a fully connected multi-layer perceptron (MLP), followed by the Softmax model to perform 3-way classification.

5-way Ensemble Model
One way to utilize multiple models is to combine uni-modal model predictions in an ensemble classifier to predict final labels. As elaborated in 3.2, 3-way textual entailment model is helpful in distinguishing three-way entailment relationship based on linguistic and semantic clues between and . To address multimodal entailment (as defined in this task 3.1), a simple relatedness measurement for visual content between and is adopted based on image pairwise similarity as a proxy of their visual entailment computed with a pre-trained CNN model (ResNet) in our approach. The same as text entailment, this is based on hypothesis and salient correlation patterns observed in this dataset that an article is relevant to a claim if the article contains images similar to the claim's images.
Hence, an ensemble approach is proposed to combine textual entailment, visual relatedness measurement and additional data specific features. More specifically, the proposed ensemble model utilizes a basic decision tree classifier with the following feature encoding to provide end-to-end five categories classification as depicted in Fig. 1.

5-way Multimodal Entailment
We consider how to obtain attended multimodal information that can effectively capture consistency and integrity of multimedia content between and . Inspired by current advance in attention techniques [60,61,46,47], we apply multiple attention mechanisms to learn the multimodal interaction between pairs of visual and textual content. Instead of local alignment approaches that model visual objects and textual words, we focus on global alignment based methods in this study that aim to map whole image and sentences into joint semantic space.
A popular framework to model the multimodal relationship is using a multi-branch attention network, typically one branch projects the image and another models the text. The similarity is measured by dot product on the normalised feature vectors. The extended MatchPyramid (elaborated in section 3.2) is applied to model high-level relevance between text pairs. In general, our end-to-end multimodal entailment architecture consists of embedding layer, text matching layer, multimodal matching layer, and classification layer. Formally, as input representations, Images pairs ( and ) are represented as the top layer (before softmax) of a pre-trained convolutional network and text pairs ( and ) are mapped into a vector ∈ R by a fixed word embedding layer initialised by Glove embeddings. denote as word embedding dimension (e.g., set to 50 for Glove-50). There is no restriction on the choice of the image encoder but the pre-trained ResNet-50 model is used in our experiments because of its simplicity.
In embedding layer, Let be the dimension of an image visual vector (i.e., = 2049 for ResNet-50) and Let and be the number of words in and respectively. Let ∈ R and ∈ R × be claim image embedding vector and word embedding matrix, respectively. Likewise, let ∈ R and ∈ R × be document image embedding vector and word embedding matrix, respectively. Each 2048-dim feature vector ( and ) is fed into a (non-trainable) linear layer to reduce the visual features from 2048 to 512 dimensional vector space in this work. For the embeddings of the text pair, self-attention (SDP) is applied (as specified in section 3.2 for both and before feeding into a separate GRU layer to obtain both of their context sequence representation ( _ ∈ R × and _ ∈ R × ) and corresponding global representation (i.e., final state), denoted by _ ∈ R and _ ∈ R . denotes the GRU output dimension.
In subsequent text matching layer, the same pipeline (as specified above for extended Match-Pyramid) is applied in attempt to model the high level relevance between article content ( ) to claim text ( ) based on contextual word embedding interactions. Interaction feature matrix is calculated by the matrix dot product between _ and _ , which is then applied by deep hierarchical convolution layers in the MatchPyramid model to extract aggregated similarity feature vector _ _ ∈ R and is the output dimension of flattened feature maps. The high-level matching patterns are then fed into a multi-layer perceptron (MLP) with dropout to produce the final matching score with learnable weights.
Multimodal latent interaction features are derived in multimodal matching layer which mainly consists of visual matching layer and cross-modal attention layer. Fundamentally, multimodal matching layer aims to find potential relevance of document visual vector ( ) to claim vectors of either visual or text context or both ( and _ ), and are hence critical for predicting multimodal entailment relations for target claim. The same as the process for text embeddings, our visual matching layer utilises self-attention (SDP) for image pairs embeddings ( and ) in attempt to capture important features in each image. Then, a visual similarity feature ( , = ⃦ ⃦ ⃦⃗ is computed by applying dot product to the two saliency guided image embeddings with 2 norm. This practice follows [62,63,47], which is a simple yet very efficient SoTA technique in handling variations in image illumination, view points, texture and season. It also links to feature whitening, linear discrimination analysis and image saliency. In addition, a separate image euclidean similarity feature ( , = 1 ) is computed in an attempt to measure potential adversarial images, same people in different scene, cropped image, etc [64,65].
In cross-modal attention layer, multiple attention mechanisms are applied, aiming to learn to find and attend to the maximum relevant elements on both feature importance and relational mapping among each self-attended visual and content vectors between claim and document.
Specifically, text-image interaction feature between document image and claim text is computed using same SDP attention: , Finally, in the classification layer, the outputs of two MLP layers are merged along with corresponding hypothesis multimodal representation ( ) into a combined representation with batch normalisation.
is the output of a separate MLP layer for merged (concatenation) _ and normalised _ ( ∈ R + ). Final representation are applied with dropout regularization before feeding to a softmax layer to output the probabilities of five categories.

Data statistics
Five entailment categories are balanced in both train and validation data set. As mentioned above, both claim (hypothesis) and document(premise) contains an image and a text of vary length. Optical character recognition (OCR) text extracted from image are not counted separately here and combined with corresponding claim and document text respectively. Thus, large claim word size presented in the table is due to extracted OCR text. The data details for each set are shown in Table 3. More details about the dataset and task details can be referred in [12].

Word Overlaps distribution
Word overlap is an important indication for modeling textual entailment as well as of potential data bias. Naturally, when pairing claims to evidence sentences, the word overlap ratio will be higher on average for claims with their supporting evidence. However, models relying on word-overlap will perform poorly when dealing with complexity in the real world examples (typically antonymous examples and adversarial attacks). In VITAMINC [66] and FEVER [67] Table 4. The data distribution indicates that evidential premise data in Factify have clearly high word overlap ratio than other two categories of insufficient evidence.

Image Similarity distribution
Image similarity is the most basic indicator of multimodal entailment. To empirically validate the intuition of potential similarity correlation between and (as mentioned in 3.3) in the dataset, image relatedness analysis is conducted in this section. Similar to the potential bias from data overlapping, we compute image pairwise similarity distribution with embedding space computed from pre-trained ResNet50 model over train/val set as presented in Table 5 and 6. As seen from the distribution over five categories, two text related entailment categories have clearly lower pairwise image similarity that multimodal evidential entailment categories.

Text Length Distribution
Text and OCR (text) length distributions between ( , ) in the train set are presented in Fig 2 and  and 'Support_Multimodal' tend to include slightly shorter claims. In comparison, OCR text for both claim and document in "Refute" category samples shows surprising longer length than other four categories. Motivated by the observation, we adopted text lengths as features in our ensemble model as illustrated in 3.3.

Image Domain Bias
Source bias is one of the known and common problems in machine learning dataset [68] which happens when most data samples are collected from the same source. This problem has several links which mainly include selection bias, capture bias (bias happens due to particular data collection methods), label bias and negative set bias. We are interested in probing potential source bias in Factify datset. The potential bias in images domains over five categories brought to our attention in data analysis. This is important since multimedia metadata have been proved to be valuable information and signal for fact verification [69,5,70] in real-world application such as domain/source credibility, detection of image manipulation and tampering. Image (link) domains are extracted from all document samples of train set and distributions are computed

Experimental Setting
To evaluate the performance of our baseline solutions, we use weighted average F1 for benchmarking on validation set. All the experiments are implemented on a single NVIDIA A100 GPU with up to 20 GiB RAM.

Hypothesis Only Test
We conducted hypothesis only reliance test by using hypothesis only information to train a model as baseline. This is a commonly adopted approach [71,72] in SNLI/RTE to verify the presence of data bias. The assumption is that without any premise information, this baseline is supposed to make a random guess out of the five classes. We train two models with or without images and test the resulting accuracy for each model on val set. Two models (Hypo and Hypo + ) are implemented with similar architecture that consists of a text processing component and/or ResNet embedding layer followed by two fullyconnected (FC) layers. The text processing component is used to extract the text feature from the given hypothesis. It firstly generates a sequence of word embeddings for the given claim text. The embedding sequence is then fed into a GRU [73]  Our experiments shows that the resulting accuracy and weighted F1 achieves same value in both 0.60 and both 0.64 on val set for text and text+image models respectively, implying the existence of bias in Factify dataset. The details are presented in results section. Our proposed solutions outperform the two hypothesis only baselines.

3-way text entailment
Transformer fine-tune settings: For the best entailment model, the pre-trained Big-Bird model from Huggingface and the implementation for pair-wise classification finetune was used. For our experiment the model was fine-tuned for 2 epochs, using the AdamW optimizer with learning rate 2e-5 and epsilon 1e-8 with batch size 4. Maximum sentence length was set to the mean length of the input texts namely, 1396 tokens. To train the 3-way entailment model, the 5-way data categories were converted to 3-way categories including "Support" ("Support_Multimodal"+"Support_Text"), "Refute", In-sufficient("Insufficient_Multimodal"+"Insufficient_Text"). OCR text from both and were excluded.
MatchPyramid baseline model settings: We use Glove embeddings with 50 dimension (GloVe 6B 50d) for text input and GRU output dimension is set to 50. The number of CNN layers is set to 2, each with kernel size 3 × 3, pooling size 5 × 10 and in valid mode (i.e., no padding). ReLu activation is placed between all convolutional layers. Convolution channels are set to 16 and 32 respectively. 2-layer MLP are set with hidden dimensions of 128 and 64 respectively. The maximum text length for and are set to 100 and 1000 respectively. Limited experiments are conducted including content context size of claim and document, global average pooling and convolution padding schemes.

5-way Ensemble Model
The ensemble model is implemented using sklearn's DecisionTreeClassifier class. Best pretrained transformer ("BigBird") based text entailment classifier is adopted and pairwise image similarity is computed based on ResNet-50. For image pre-processing and feature extraction, same practice introduced in 5.2 is applied. One-hot encoding function provided in Scikit-learn is utilised to converts two categorical features (i.e., URL domains of and ) as a one-hot numeric array learnt from train set. Text are pre-processed separately for BigBird model and ensemble model. No pre-processing is applied for four text length features. For the ensemble model training, we use 'best' split based on 'gini' impurity matrix as training criteria and limit the number of layers to 8 to avoid overfitting.

5-way Multimodal classification
5-way end-to-end Multimodal Entailment model Multimodal is implemented with Keras and tensorflow(v2.4) with Adam optimiser with an adaptive learning scheduler. The initial learning rate and weight decay are both set to 0.0001. Batch size is 32 and maximum number of training epochs is set to 80. Optimal parameters and settings from MatchPyramid baseline model experiments are applied. Checkpoint callback is used to save best model that achieves best validation accuracy. ReLu activation is applied to all convolution layers and fully-connected layers. The uniform He initialization ("he_uniform") is used for all ReLU layers. Same settings (layer size, hidden dim, activation, etc) are applied for three separate MLP layers. Few parameters and architectures are experimented including vary lengths of claim and document content, MLP layer size (1-2), hidden layer dimensions of MLPs (64,256,512,768,1024), merge strategies (concatenation and multiplication) of three MLPs output for classification layer. Ablation study is conducted by removing individual sub-components including hypothesis MLP ( ), document crossmodal interaction( ,

3-way text entailment
The results of the 3-way text entailment models are presented in Table 7. To validate our model choice, we evaluate few SoTA pre-trained transformer models, including BERT, RoBerta, BigBird and LongFormer. The best performing models are BigBird and LongFormer with the overall winner being BigBird because of the slightly better results and smaller input context size required (1396 vs 1484 respectively).
For the architecture of extended MatchPyramid baseline, we have experimented with different parameters such as longer context length in (including 1500, 2000, 3000), Glove model with 300 dimension, larger GRU output dimension of 300, various pooling size ( [3,10]), etc, none of these attempts provide major improvement. Overall, our baseline implementation with self-attention and Glove-50d based contextual representation learning achieve optimal performance, which is competitive to large transformer model based approaches as presented in Table 7.

5-way classification
The results of 5-way and model on val set are presented in Table  8. Four of our baseline methods outperform all baseline models proposed by task organiser as reported in [12]. The result of best baseline model (Multimodal ) in Factify data paper is presented in the table 1 . Unsurprisingly, our ensemble model achieved best results on val set with 0.77 F1 which is 8% higher than the results of model. The experiment results demonstrate a large performance gain with the large pre-trained text entailment model which works effectively on long paragraphs and contribute the most towards predicting final 5-way categories. This is particularly obvious for "Refute" label, the samples in which are mostly relying on text based inference. It is not surprising that the useful features incorporated from the heuristics and bias learned from the dataset have proved to be effective for this multimodal prediction task.
It was found that differentiating between "Insufficient_Multimodal" and "Support_Text" or between "Insufficient_Text" and "Support_Multimodal" was the most challenging task without relying data specific features. In other words, when a sample contains supporting document text for the claim but the image is irrelevant, our model has low confidence in predicting the label as "Support_Text" or "Insufficient_Multimodal". Likewise, when document image is relevant to claim image about same information context but document text is irrelevant, our model has low confidence in predicting the correct label. The decision is highly dependent on the annotation bias. From all the labels, the "Refute" label is the most distinguishable category and highly dependent on the text. The performance is highly consistent among all our models and participant systems in this competition (as seen in leaderboard 10). This is possibly mainly attributed to the articles samples selected from very few fact checking sources that have highly differentiable linguistic clues (typically high frequent negative words used and same verdict sentences frequently appeared in this category such as "The claim is false").

Competition Result
Final test set results and competition leaderboard are presented in Table 9 and 10 respectively. Our best model ("Ensemble") outperform all competition systems and best baseline models. Test result of model achieved 0.77 avg. F1 which is the same as the result on val set and 10% higher than the result of .

Conclusion
We described our participation in the Multimodal Fact Verification Factify Challenge with the implementation of two proposed baseline solutions including an ensemble model and an end-to-end multimodal entailment model. Ensemble model based system outperform the endto-end model on val set and test set. The best performing model in this competition combines results of 3-way text entailment classifier, visual similarity with a pre-trained CNN model and heuristics learnt from the dataset. Multimodal fusion technique is explored in this paper to model interaction between different modalities (i.e., text and image) in claim and document pairs and combines information from them to learn multimodal entailment relationship end-to-end. We found that multimodal entailment based system suffer from overfitting. Apart from limited train size and identified data bias, our experiments suggest that fine-grained image and text interaction model need to be explored further. We found that the ambiguous labels in Factify dataset undermines the performance of our deep learning architecture. Creating a dataset for a complex real-word multimodal NLP problems particularly natural language inference as multimodal verification has raised emergent challenges [74,75] and indeed a cumbersome task, and we appreciate the work by the Factify organizers, yet, a more elaborate and unbiased dataset along with well defined annotation criterion should make this dataset more suitable for benchmark. More effort is required to tackle the dataset challenge of minimising hypotheses from human annotators and make dataset better reflecting real-world challenges. As an emergent research field, we hope our extensive data analysis and proposed baseline solutions can inspire further work.