Our proposed model DISARM, as depicted in Fig. REF , is based on a fusion of the textual and the visual modalities, explicitly enriched via contextualised representations by leveraging CLIP [1]}. We chose CLIP as a preferred encoder module for contextualization, due to its impressive zero-shot multimodal embedding capabilities. At first, valid entities are extracted automatically, as part of the process of creating training/validation sets. Then, for each meme, we first obtain the contextualized-entity (CE) representation by fusing the CLIP-encoded context and the entity representation. CE is then fused with BERT-based [2]} embedded-harmfulness (EH) encoding fine-tuned on the OCR-extracted text and entities as inputs. We call the resulting fusion output a contextualized-text (CT) representation. CT is then fused with the contextualized-image (CI) representation, obtained using the CLIP encoder for the image. We, henceforth, refer to the resulting enriched representation as the contextualized multimodal (CMM) representation.
We modify the multimodal low-rank bi-linear pooling [3]} to fuse the input representation into a joint space.
