Multimodal fusion mainly refers to the comprehensive processing of multimodal data by computer, which is responsible for fusing the information of each mode to perform target prediction [1]}.
Tensor Fusion Network (TFN) [2]}is a multimodal network for features fusion through matrix operation to directly fuse the three features vectors of the data with three modes (such as text, image and audio). However, since TFN calculates the correlation between the elements of different modes through the tensor outer product between modes, it will greatly increase the dimension of features tensor and result in a too large model that is difficult to train.
Low-rank Multimodal Fusion [3]} uses a low rank matrix to decompose the weight, and hence the TFN process is changed into a single linear transformation of each mode. Then the received multi-dimensional point by Low-rank Multimodal Fusion can be regarded as the sum of multiple low rank vectors, and thus the number of parameters in the model is reduced. Although Low-rank Multimodal Fusion is an upgrade of TFN, once the features are too long, it is still easy to explode parameters.
Multimodal Adversarial Representation Network [4]} adds a dual discriminator countermeasure network based on multimodal fusion (ordinary attention fusion), which captures dynamic commonness and invariance respectively.
Multimodal Bottleneck Transformer [5]} uses a shared token between two Transformer, so that this token becomes a communication bottleneck of different modes to save computational attention. In this way, multimodal interaction can be limited to several shared tokens.
Compared with the above researches, we pay more attention to multimodal time-series events, and the above researches can also be regarded as special cases of multimodal time-series events.
