Data of different modalities generally convey complimentary but heterogeneous information, and a more discriminative representation is often preferred by combining multiple data modalities like the RGB and infrared features. However in reality, obtaining both data channels is challenging due to many limitations. For example, the RGB surveillance cameras are often restricted from private spaces, which is in conflict with the need of abnormal activity detection for personal security. As a result, using partial data channels to build a full representation of multi-modalities is clearly desired. In this paper, we propose a novel Partial-modal Generative Adversarial Networks (PM-GANs) that learns a full-modal representation using data from only partial modalities. The full representation is achieved by a generated representation in place of the missing data channel. Extensive experiments are conducted to verify the performance of our proposed method on action recognition, compared with four state-of-the-art methods. Meanwhile, a new Infrared-Visible Dataset for action recognition is introduced, and will be the first publicly available action dataset that contains paired infrared and visible spectrum. (The dataset will be available at http: www.escience.cn people gaochenqiang Publications.html).
Recently, increasing efforts have been devoted to infrared action recognition @cite_29 . Corresponding to the classical methods employed in visible spectrum, spatiaotemporal representation for human action repretitive action recogtnition is also used under thermal imaging scenarios @cite_41 . The combination of both visible and thermal imaging to improve human silhouette detection is also introduced by Han @cite_50 . However, the scenario has not been studied where infrared data is available while the RGB channel is missing. The scenario has great potential in real-world of protecting privacy while benefiting the task of action recognition, and is meaningful to both the study of pattern recognition and the welfare of the community at large. Therefore, we are motivated to dedicate to improving the situation by constructing a robust and discriminative partial-modal representations, and to specify action recognition as the case in this paper.
@cite_41 @cite_29 @cite_50
2114486396 2473890477 1970579918
In this paper, we investigate human repetitive activity properties from thermal infrared imagery, where human motion can be easily detected from the background regardless of lighting conditions and colors of the human surfaces and backgrounds. We employ an efficient spatiotemporal representation for human repetitive activity recognition, which represents human motion sequence in a single image while preserving some temporal information. A statistical approach is used to extract features for activity recognition. Experimental results show that the proposed approach achieves good performance for human repetitive activity recognition. Abstract Action recognition (AR) is one of the most important tasks in video analysis and computer vision. Recently a large number of related methods have been proposed. While most of these methods are investigated on AR datasets collected from the visible spectrum, the AR problem under infrared scenarios still has not attracted much attention. There is even few public infrared datasets available for supporting the fundamental evaluation requirements of this research. To this issue, this work aims to emphasize the importance of the infrared AR problem in applications and arouse researchers' attention on this task. Specifically, we construct a new Inf rared A ction R ecognition ( InfAR ) dataset captured at different times, including in summer and winter, and explore how discriminable actions in our InfAR dataset are with the state-of-the-art pipelines based on low-level features and deep convolutional neural network (CNN), respectively. Our results reveal: (1) In all, dense trajectory feature can achieve the best performance while the appearance features, e.g., HOG, have relatively poorer performance; (2) the encoding method of vector of locally aggregated descriptors is evidently better than that of the widely-used Fisher Vector; (3) the late fusion facilitates a better performance than early fusion; (4) action videos captured in winter is more discriminable than in summer; (5) compared to appearance information, the motion information is more essential for infrared action recognition and utilizing this information through deep CNN can improve greatly the performance. The best performance achieved on our dataset is 76.66 (Average Precision), leaving a reasonable space for further exploring the insights underlying such type of infrared AR problem and accordingly designing proper techniques to further promote the performance on this specifically constructed InfAR dataset. We approach the task of human silhouette extraction from color and thermal image sequences using automatic image registration. Image registration between color and thermal images is a challenging problem due to the difficulties associated with finding correspondence. However, the moving people in a static scene provide cues to address this problem. In this paper, we propose a hierarchical scheme to automatically find the correspondence between the preliminary human silhouettes extracted from synchronous color and thermal image sequences for image registration. Next, we discuss strategies for probabilistically combining cues from registered color and thermal images for improved human silhouette detection. It is shown that the proposed approach achieves good results for image registration and human silhouette extraction. Experimental results also show a comparison of various sensor fusion strategies and demonstrate the improvement in performance over non-fused cases for human silhouette extraction.