In general terms, a modality refers to the way in which something happens or is experienced [1]}.
To our best knowledge, many existing works have demonstrated that Neural Network can achieve an excellent result in single modality processing such as image classification [2]}, speech synthesis [3]}, natural language processing [4]}.
In the field of data, multimodal is used to represent different forms of data, or different formats of the same form, which generally represents text, picture, audio and video [5]}, [6]}.
Hence, multimodal data processing have attracted a wide attention from the academia, especially for multimodal fusion which is one of the original topics in multimodal machine learning [1]}. Neural Networks is expected to tackle the multimodal fusion problem [8]} and has been used extensively to fuse information for text, image and audio [9]}, [10]}, gesture recognition [11]}, and video or image description generation [12]}, [13]}, since the earliest investigation of AVSR [14]}.
However, almost of all these studies focus on text, images or speech modes rather than multimodal time-series which is a critical ingredient across many domains, so how to effectively process multimodal data still need further study.
Many methods have been launched to process simple single mode time-series data [15]}, [16]}, which have achieved the best result in their respective field. But they have no way to directly use multimodal time-series data, for example multisensor data, medical time-series data.
