The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably,  modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequential learning. FMT inherently models the intramodal and intermodal (involving two or more modalities) dynamics within its multimodal input in a factorized manner. The proposed factorization allows for increasing the number of self-attentions to better model the multimodal phenomena at hand; without encountering difficulties during training (e.g. overfitting) even on relatively low-resource setups. All the attention mechanisms within FMT have a full time-domain receptive field which allows them to asynchronously capture long-range multimodal dynamics. In our experiments we focus on datasets that contain the three commonly studied modalities of language, vision and acoustic. We perform a wide range of experiments, spanning across 3 well-studied datasets and 21 distinct labels. FMT shows superior performance over previously proposed models, setting new state of the art in  the studied datasets. In many naturally occurring scenarios, our perception of the world is multimodal. For example, consider multimodal language (face-to-face communication), where modalities of language, vision and acoustic are seamlessly used together for communicative intent (Kottur et al., 2019) . Such scenarios are widespread in everyday life, where continuous sensory perceptions form multimodal sequential data. Each modality within multimodal data exhibits exclusive intramodal dynamics, and presents a unique source of information. Modalities are not fully independent of each other. Relations across two (bimodal) or more (trimodal, . . . ) of them form intermodal dynamics; often asynchronous spatio-temporal dynamics which bind modalities together . Learning from multimodal sequential data has been an active, yet challenging research area within the field of machine learning (Baltrušaitis et al., 2018) . Various approaches relying on graphical models or RNNs have been proposed for multimodal sequential learning. Transformer models are a new class of neural models that rely on a carefully designed non-recurrent architecture for sequential modeling (Vaswani et al., 2017) . Their superior performance is attributed to a self-attention mechanism, which is uniquely capable of highlighting related information across a sequence. This self-attention is a particularly appealing mechanism for multimodal sequential learning, as it can be modified into a strong neural component for finding relations between different modalities (the cornerstone of this paper). In practice, numerous such relations may simultaneously exist within multimodal data, which would require increasing the number of attention units (i.e. heads). Increasing the number of attentions in an efficient and semantically meaningful way inside a transformer model, can boost the performance in modeling multimodal sequential data. In this paper, we present a new transformer model for multimodal sequential learning, called Factorized Multimodal Transformer (FMT) . FMT is capable of modeling asynchronous intramodal and intermodal dynamics in an efficient manner, within one single transformer network. It does so by specifically accounting for possible sets of interactions between modalities (i.e. factorizing based on combinations) in a Factorized Multimodal Self-attention (FMS) unit. We evaluate the performance of FMT on multimodal language: a challenging type of multimodal data which exhibits idiosyncratic and asynchronous spatio-temporal relations across language, vision and acoustic modalities. FMT is compared to previously proposed approaches for multimodal sequential learning over multimodal sentiment analysis (CMU-MOSI) (Zadeh et al., 2016) , multimodal emotion recognition (IEMOCAP) (Busso et al., 2008) , and multimodal personality traits recognition (POM) (Park et al., 2014) . The results of sentiment analysis experiments on CMU-MOSI dataset are presented in Table 1 . FMT achieves superior performance than the previously proposed models for multimodal sentiment analysis. We use two approaches for calculating BA and F1 based on negative vs. non-negative sentiment (Zadeh et al., 2018b) on the left side of /, and negative vs. positive (Tsai et al., 2019) on the right side. MAE and Corr are also reported. For multimodal emotion recognition, experiments on IEMOCAP are reported in Table 2 . The performance of FMT is superior than other baselines for multimodal emotion recognition (with the exception of Happy emotion). The results of experiments for personality traits recognition on POM dataset are reported in Table 3 . We report MA5 and MA7, depending on the label. FMT outperforms baselines across all personality traits. We study the importance of the factorization in FMT. We first remove the unimodal, bimodal and trimodal attentions from the FMT model, resulting in 3 alternative implementations of FMT. demonstrates the results of this ablation experiment over CMU-MOSI dataset. Furthermore, we use only one modality as input for FMT, to understand the importance of each modality (all other factors removed). We also replace the summarization networks with simple vector addition operation. All factors, modalities, and summarization components are needed for achieving best performance. We also perform experiments to understand the effect of number of FMT units within each MTL. Table 5 shows the performance trend for different number of FMT units. The model with 6 number of FMS (42 attentions in total) achieves the highest performance (6 is also the highest number we experimented with). Tsai et al. (2019) reports the best performance for CMU-MOSI dataset is achieved when using 40 attentions per cross-modal transformer (3 of each, therefore 120 attention, without counting the subsequent unimodal transformers). FMT uses fewer number of attentions than MulT, yet achieves better performance. We also experiment with number of heads for original transformer model (Vaswani et al., 2017) and compare to FMT (Appendix A.3). In this paper, we presented the Factorized Multimodal Transformer (FMT) model for multimodal sequential learning. Using a Factorized Multimodal Self-attention (FMS) within each Multimodal Transformer Layer (MTL), FMT is able to model the intra-model and inter-modal dynamics within asynchronous multimodal sequences. We compared the performance of FMT to baselines approaches over 3 publicly available datasets for multimodal sentiment analysis (CMU-MOSI, 1 label), emotion recognition (IEMOCAP, 4 labels) and personality traits recognition (POM, 16 labels). Overall, FMT achieved superior performance than previously proposed models across the studied datasets. A APPENDIX
A multimodal transformer for multimodal sequential learning, with strong empirical results on multimodal language metrics such as multimodal sentiment analysis, emotion recognition and personality traits recognition. 
{"src": "scitldr-AIC_train", "paper_id": "BJxD11HFDS"}