Action monitoring in a home environment provides important information for health monitoring and may serve as input into a smart home environment. Visual analysis using cameras can recognise actions in a complex scene, such as someones living room. However, although there the huge potential benefits and importance, specifically for health, cameras are not widely accepted because of privacy concerns. This paper recognises human activities using a sensor that retains privacy. The sensor is not only different by being thermal, but it is also of low resolution: 8x8 pixels. The combination of the thermal imaging, and the low spatial resolution ensures the privacy of individuals. We present an approach to recognise daily activities using this sensor based on a discrete cosine transform. We evaluate the proposed method on a state-of-the-art dataset and experimentally confirm that our approach outperforms the baseline method. We also introduce a new dataset, and evaluate the method on it. Here we show that the sensor is considered better at detecting the occurrence of falls and Activities of Daily Living. Our method achieves an overall accuracy of 87.50 across 7 activities with a fall detection sensitivity of 100 and specificity of 99.21 .
An infrared sensor array is a device composed of a small number of discrete infrared sensors. It represents the spatial distribution of temperature as a low-resolution image. Unlike colour cameras, infrared sensor arrays only capture the shape of the human body, therefore making individual identification harder. Additionally, the low spatial resolution also makes identification of individuals difficult. As this is more comfortable for users, it becomes more acceptable for installation in residential environments. Such infrared sensor arrays can be applied in many scenarios. A 4x4 sensor array has been used to recognise hand motion directions @cite_5 , although the extremely low resolution of this sensor renders it unsuitable for more complex visual tasks. A 8x8 pixel sensor array has been successfully used to detect, count and track people indoors @cite_6 . Human movements has also be inferred by using the subject's location and moving trajectory using a 16x16 sensor array @cite_10 . Most recently, a multi-sensor system has been designed for human movement detection and activity recognition @cite_1 , which our method will be compared against.
@cite_1 @cite_5 @cite_10 @cite_6
957840212 2601917592 2180097222 2287367556 2043167551 2114486396 2051071899 2067183916 2473890477 1978993121 2000814733 1982297923 2787943737 1975851618 2771763340 2753791803 2312470762 2741620214 2417485407 2797030127
We propose a human body tracking method using a far-infrared sensor array. A far-infrared sensor array captures the spatial distribution of temperature as a low-resolution image. Since it is difficult to identify a person from the low-resolution thermal image, we can avoid privacy issues. Therefore, it is expected to be applied for the analysis of human behaviors in various places. However, it is difficult to accurately track humans because of the lack of information sufficient to describe the feature of the target human body in the low-resolution thermal image. In order to solve this problem, we propose a thermo-spatial sensitive histogram suitable to represent the target in the low-resolution thermal image. Unlike the conventional histograms, in case of the thermo-spatial sensitive histogram, a voting value is weighted depending on the distance to the target’s position and the difference from the target’s temperature. This histogram allows the accurate tracking by representing the target with multiple histograms and reducing the influence of the background pixels. Based on this histogram, the proposed method tracks humans robustly to occlusions, pose variations, and background clutters. We demonstrate the effectiveness of the method through an experiment using various image sequences. Low Resolution Thermal Array Sensors are widely used in several applications in indoor environments. In particular, one of these cheap, small and unobtrusive sensors provides a low-resolution thermal image of the environment and, unlike cameras; it is capable to detect human heat emission even in dark rooms. The obtained thermal data can be used to monitor older seniors while they are performing daily activities at home, to detect critical situations such as falls. Most of the studies in activity recognition using Thermal Array Sensors require human detection techniques to recognize humans passing in the sensor field of view. This paper aims to improve the accuracy of the algorithms used so far by considering the temperature environment variation. This method leverages an adaptive background estimation and a noise removal technique based on Kalman Filter. In order to properly validate the system, a novel installation of a single sensor has been implemented in a smart environment: the obtained results show an improvement in human detection accuracy with respect to the state of the art, especially in case of disturbed environments. Indoor tracking has all-pervasive applications beyond mere surveillance, for example in education, health monitoring, marketing, energy management and so on. Image and video based tracking systems are intrusive. Thermal array sensors on the other hand can provide coarse-grained tracking while preserving privacy of the subjects. The goal of the project is to facilitate motion detection and group proxemics modeling using an 8 x 8 infrared sensor array. Each of the 8 x 8 pixels is a temperature reading in Fahrenheit. We refer to each 8 x 8 matrix as a scene. We collected approximately 902 scenes with different configurations of human groups and different walking directions. We infer direction of motion of a subject across a set of scenes as left-to-right, right-to-left, up-to-down and down-to-up using cross-correlation analysis. We used features from connected component analysis of each background subtracted scene and performed Support Vector Machine classification to estimate number of instances of human subjects in the scene. In this paper we present our work towards a hand gesture recognition system realised with a passive thermal infrared sensor array. In contrast with the majority of recent research activities into gesture recognition, which focus on the complex analysis of video sequences, our approach shows that the functionality of a simple pyroelectric movement sensor can be expanded to detect differing hand gestures at short range. We show that blob detection from a hand waving over a 16 element passive infrared sensor array provides sufficient information to discriminate four directions of hand stroke. This sensor system is unique and lends itself to low cost, low profile and low power applications. Keywords-touchless input device; dynamic hand gesture; infrared motion sensor; infrared sensor array; pyroelectricity; Short wave infrared (SWIR) is an emerging imaging modality in surveillance applications. It is able to capture clear long range images of a subject in harsh atmospheric conditions and at night time. However, matching SWIR images against a gallery of color images is a very challenging task. The photometric properties of images in these two spectral bands are highly distinct. This work presents a novel cross-spectral face recognition scheme that encodes images filtered with a bank of Gabor filters followed by three local operators: Simplified Weber Local Descriptor, Local Binary Pattern, and Generalized Local Binary Pattern. Both magnitude and phase of filtered images are encoded. Matching encoded face images is performed by using a symmetric I-divergence. We quantify the verification and identification performance of the cross-spectral matcher on two multispectral face datasets. In the first dataset (PRE-TINDERS), both SWIR and visible gallery images are captured at a close distance (about 2 meters). In the second dataset (TINDERS), the probe SWIR images are collected at longer ranges (50 and 106 meters). The results on PRE-TINDERS dataset form a baseline for matching long range data. We also demonstrate the capability of the proposed approach by comparing its performance with the performance of Faceit G8, a commercial face recognition engine distributed by L1. The results show that the designed method outperforms Faceit G8 in terms of verification and identification rates on both datasets. In this paper, we investigate human repetitive activity properties from thermal infrared imagery, where human motion can be easily detected from the background regardless of lighting conditions and colors of the human surfaces and backgrounds. We employ an efficient spatiotemporal representation for human repetitive activity recognition, which represents human motion sequence in a single image while preserving some temporal information. A statistical approach is used to extract features for activity recognition. Experimental results show that the proposed approach achieves good performance for human repetitive activity recognition. The interest in automatic surveillance and monitoring systems has been growing over the last years due to increasing demands for security and law enforcement applications. Although, automatic surveillance systems have reached a significant level of maturity with some practical success, it still remains a challenging problem due to large variation in illumination conditions. Recognition based only on the visual spectrum remains limited in uncontrolled operating environments such as outdoor situations and low illumination conditions. In the last years, as a result of the development of low-cost infrared cameras, night vision systems have gained more and more interest, making infrared (IR) imagery as a viable alternative to visible imaging in the search for a robust and practical identification system. Recently, some researchers have proposed the fusion of data recorded by an IR sensor and a visible camera in order to produce information otherwise not obtainable by viewing the sensor outputs separately. In this article, we propose the application of finite mixtures of multidimensional asymmetric generalized Gaussian distributions for different challenging tasks involving IR images. The advantage of the considered model is that it has the required flexibility to fit different shapes of observed non-Gaussian and asymmetric data. In particular, we present a highly efficient expectation-maximization (EM) algorithm, based on minimum message length (MML) formulation, for the unsupervised learning of the proposed model's parameters. In addition, we study its performance in two interesting applications namely pedestrian detection and multiple target tracking. Furthermore, we examine whether fusion of visual and thermal images can increase the overall performance of surveillance systems. The increased power densities of multi-core processors and the variations within and across workloads lead to runtime thermal hot spots locations of which change across time and space. Thermal hot spots increase leakage, deteriorate timing, and reduce the mean time to failure. To manage runtime thermal variations, circuit designers embed within-die thermal sensors that acquire temperatures at few selected locations. The acquired temperatures are then used to guide runtime thermal management techniques. The capabilities of these techniques are essentially bounded by the spatial thermal resolution of the sensor measurements. In this paper we characterize temperature signals of real processors and demonstrate that on-chip thermal gradients lead to sparse signals in the frequency domain. We exploit this observation to (1) devise thermal sensor allocation techniques, and (2) devise signal reconstruction techniques that fully characterize the thermal status of the processor using the limited number of measurements from the thermal sensors. To verify the accuracy of our methods, we compare our temperature characterization results against thermal measurements acquired from a state-of-the-art infrared camera that captures the mid-band infrared emissions from the back of the die of a 45 nm dual-core processor. Our results show that our techniques are capable of accurately characterizing the temperatures of real processors. Abstract Action recognition (AR) is one of the most important tasks in video analysis and computer vision. Recently a large number of related methods have been proposed. While most of these methods are investigated on AR datasets collected from the visible spectrum, the AR problem under infrared scenarios still has not attracted much attention. There is even few public infrared datasets available for supporting the fundamental evaluation requirements of this research. To this issue, this work aims to emphasize the importance of the infrared AR problem in applications and arouse researchers' attention on this task. Specifically, we construct a new Inf rared A ction R ecognition ( InfAR ) dataset captured at different times, including in summer and winter, and explore how discriminable actions in our InfAR dataset are with the state-of-the-art pipelines based on low-level features and deep convolutional neural network (CNN), respectively. Our results reveal: (1) In all, dense trajectory feature can achieve the best performance while the appearance features, e.g., HOG, have relatively poorer performance; (2) the encoding method of vector of locally aggregated descriptors is evidently better than that of the widely-used Fisher Vector; (3) the late fusion facilitates a better performance than early fusion; (4) action videos captured in winter is more discriminable than in summer; (5) compared to appearance information, the motion information is more essential for infrared action recognition and utilizing this information through deep CNN can improve greatly the performance. The best performance achieved on our dataset is 76.66 (Average Precision), leaving a reasonable space for further exploring the insights underlying such type of infrared AR problem and accordingly designing proper techniques to further promote the performance on this specifically constructed InfAR dataset. The robust detection of small targets is one of the key techniques in infrared search and tracking applications. A novel small target detection method in a single infrared image is proposed in this paper. Initially, the traditional infrared image model is generalized to a new infrared patch-image model using local patch construction. Then, because of the non-local self-correlation property of the infrared background image, based on the new model small target detection is formulated as an optimization problem of recovering low-rank and sparse matrices, which is effectively solved using stable principle component pursuit. Finally, a simple adaptive segmentation method is used to segment the target image and the segmentation result can be refined by post-processing. Extensive synthetic and real data experiments show that under different clutter backgrounds the proposed method not only works more stably for different target sizes and signal-to-clutter ratio values, but also has better detection performance compared with conventional baseline methods. In this paper, a robust approach using a shape distribution histogram (SDH) feature and modified sparse representation classification (MSRC) for pedestrian detection in thermal infrared imagery is proposed. In this framework, the candidate regions that are more likely to contain the pedestrians are first detected based on the Contour Saliency Map. Then distances between random points on the thinned contour map of objects in the candidate regions are applied to acquire the SDH feature. SDH is a robust and discriminative feature which can precisely describe the pedestrian characteristics. Finally, a robust MSRC classifier which has high accuracy is used to recognize the true pedestrians. Experiments are conducted over the OSU thermal pedestrian database by comparing with other algorithms. The proposed method shows an excellent performance in detecting pedestrians. HighlightsRobust pedestrian detection using the SDH feature and MSRC classifier in infrared imagery.Discriminative SDH feature can precisely describe the pedestrian characteristic.Robust MSRC classifier is used to classify the pedestrians and non-pedestrians. In this paper we present zSense, which provides greater input expressivity for spatially limited devices such as smart wearables through a shallow depth gesture recognition system using non-focused infrared sensors. To achieve this, we introduce a novel Non-linear Spatial Sampling (NSS) technique that significantly cuts down the number of required infrared sensors and emitters. These can be arranged in many different configurations; for example, number of sensor emitter units can be as minimal as one sensor and two emitters. We implemented different configurations of zSense on smart wearables such as smartwatches, smartglasses and smart rings. These configurations naturally fit into the flat or curved surfaces of such devices, providing a wide scope of zSense enabled application scenarios. Our evaluations reported over 94.8 gesture recognition accuracy across all configurations. Recently, infrared human action recognition has attracted increasing attention for it has many advantages over visible light, that is, being robust to illumination change and shadows. However, the infrared action data is limited until now, which degrades the performance of infrared action recognition. Motivated by the idea of transfer learning, an infrared human action recognition framework using auxiliary data from visible light is proposed to solve the problem of limited infrared action data. In the proposed framework, we first construct a novel Cross-Dataset Feature Alignment and Generalization (CDFAG) framework to map the infrared data and visible light data into a common feature space, where Kernel Manifold Alignment (KEMA) and a dual aligned-to-generalized encoders (AGE) model are employed to represent the feature. Then, a support vector machine (SVM) is trained, using both the infrared data and visible light data, and can classify the features derived from infrared data. The proposed method is evaluated on InfAR, which is a publicly available infrared human action dataset. To build up auxiliary data, we set up a novel visible light action dataset XD145. Experimental results show that the proposed method can achieve state-of-the-art performance compared with several transfer learning and domain adaptation methods. Interactions between a 60-GHz microstrip patch antenna array designed for off-body communications and the human body are investigated numerically and experimentally. First, the array is characterized in free space and on a homogeneous skin-equivalent phantom in terms of reflection coefficient, radiation pattern, and antenna efficiency. Second, a multiphysics dosimetry technique is proposed and implemented to determine the specific absorption rate (SAR) and incident power density (IPD) from the heating dynamics measured on an experimental phantom using a high-resolution infrared (IR) camera. The SAR and IPD are found by fitting the analytical solution of the bio-heat transfer equation to the measured heating dynamics. The experimental and numerical results are in a very good agreement. They demonstrate that for the considered scenario the impact of the body on the antenna characteristics is almost negligible, and even relatively high radiated powers (up to 550 mW) result in exposure levels that are below international exposure limits. This paper presents new approach for unobtrusive indoor fall detection by an IR thermal array sensor. Unlike existing methods that run fall detection at server and require high communication and processing rates, we perform fall detection within the sensor node by a computationally inexpensive algorithm that signals the server only when a fall occurs. Experiments with prototype design show that such formulation provides robust and real-time fall detection even in a noisy environment. Laser devices have received increasing attention in numerous computer-aided applications such as automatic control, 3D modeling and virtual reality. In this paper, aiming at people counting, we propose a novel people detection and tracking method based on the multipoint infrared laser, which can further facilitate intelligent scene modeling and analysis. In our method, a camera with the infrared lens filter is utilized to capture the monitored scene where an array of infrared spots is produced by the multipoint infrared laser. We build a spatial background model based on locations of spots. Pedestrians are detected by clustering of foreground spots. Then, our method tracks and counts the detected pedestrians via inferring the forward–backward motion consistency. Both quantitative and qualitative evaluation and comparison are conducted, and the experimental results demonstrate that the proposed method achieves excellent performance in challenging scenarios. We consider methods to address the optical feature-aided remote sensing tracking problem for vehicles in a challenging environment. Our approach is to apply the dynamic data driven application systems computing paradigm to implement control of an adaptive sensor. This adaptive sensor acquires a panchromatic image while simultaneously allowing the collection of visible-near infrared spectral data at specified pixels. This sensor holds the promise of delivering the increased accuracy of targeted spectral sensing without the enormous data volume of full spectral images. The target of interest is optimally imaged by the sensor based on the target's forecasted location and motion relative to the extracted content of the background. Background context is both extracted from the image and created from the OpenStreetMap road network. We describe the implementation of the tracking framework and testing of some of the components using simulated imagery created with the digital imaging and remote sensing image generation model. The Gaussian sum filter is employed to solve the data assimilation problem by forming a multimodel forecasting set that is used to increase the robustness and flexibility of tracking. For feature matching, we create an efficient sampling strategy that is informed by the viewing conditions to adaptively determine which pixels to measure spectrally in order to distinguish between different targets using a spectral distance measure. Multispectral images that combine visual-optical (VIS) and infrared (IR) image information are a promising source of data for automatic person detection. Especially in automotive or surveillance applications, challenging conditions such as insufficient illumination or large distances between camera and object occur regularly and can affect image quality. This leads to weak image contrast or low object resolution. In order to detect persons under such conditions, we apply deep learning for effectively fusing the VIS and IR information in multispectral images. We present a novel multispectral Region Proposal Network (RPN) that is built up on the pre-trained very deep convolutional network VGG-16. The proposals of this network are further evaluated using a Boosted Decision Trees classifier in order to reduce potential false positive detections. With a log-average miss rate of 29:83 on the reasonable test set of the KAIST Multispectral Pedestrian Detection Benchmark, we improve the current state-of-the-art by about 18 . The remarkable accessibility of modern flying robots makes them an attractive platform for environmental sensing. However, low cost and ease of use are currently incompatible with large payloads, severely limiting the choice of sensor and ultimately modality. This paper describes the design of a system for using a small infrared thermometer to estimate the surface temperature over an area that is large compared to the area measured by the sensor, by mounting it on a flying robot. We leverage a priori knowledge about the spatial statistics of the phenomena under measure in order to plan an informative sampling path, fusing observations by Gaussian process regression. Our approach is designed to be evaluated in an indoor testbed, in which a quadrotor, in cooperation with simulated static sensing nodes, estimates the spatial distribution of surface temperature over a controlled thermal gradient. We perform extensive systematic experimentation both in simulation and our real-world testbed environment, with our algorithm estimating surface temperature to an accuracy of up to 2.1 °C over a 16 m2 area ranging in value from 25–65 °C. Data of different modalities generally convey complimentary but heterogeneous information, and a more discriminative representation is often preferred by combining multiple data modalities like the RGB and infrared features. However in reality, obtaining both data channels is challenging due to many limitations. For example, the RGB surveillance cameras are often restricted from private spaces, which is in conflict with the need of abnormal activity detection for personal security. As a result, using partial data channels to build a full representation of multi-modalities is clearly desired. In this paper, we propose a novel Partial-modal Generative Adversarial Networks (PM-GANs) that learns a full-modal representation using data from only partial modalities. The full representation is achieved by a generated representation in place of the missing data channel. Extensive experiments are conducted to verify the performance of our proposed method on action recognition, compared with four state-of-the-art methods. Meanwhile, a new Infrared-Visible Dataset for action recognition is introduced, and will be the first publicly available action dataset that contains paired infrared and visible spectrum.