Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> This paper presented a novel approach for gait recognition by analyzing the dynamics of walking patterns.Inspired by intuitive inference,a dynamic weight mask(DWM) is created by the variance of averaged gait energy images(GEIs) representing each class.A dynamics and shape enhanced gait representation called EGEI is obtained from DWM.To increase discriminative information an ensemble of Gabor kernels is convolved with EGEIs.Gabor gait is represented in the lower dimensional space using discriminative common vectors analysis.The proposed algorithm was tested on the USF HumanID Database.The experimental results demonstrate that each step in the proposed algorithm is effective and the recognition performance is improved. <s> BIB001 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> We introduce in this paper a novel method for employing image-based rendering to extend the range of applicability of human motion and gait recognition systems. Much work has been done in the field of human motion and gait recognition, and many interesting methods for detecting and classifying motion have been developed. However, systems that can robustly recognize human behavior in real-world contexts have yet to be developed. A significant reason for this is that the activities of humans in typical settings are unconstrained in terms of the motion path. People are free to move throughout the area of interest in any direction they like. While there have been many good classification systems developed in this domain, the majority of these systems have used a single camera providing input to a training-based learning method. Methods that rely on a single camera are implicitly view-dependent. In practice, the classification accuracy of these systems often becomes increasingly poor as the angle between the camera and the direction of motion varies away from the training view angle. As a result, these methods have limited real-world applications, since it is often impossible to limit the direction of motion of people so rigidly. We demonstrate the use of image-based rendering to adapt the input to meet the needs of the classifier by automatically constructing the proper view (image), that matches the training view, from a combination of arbitrary views taken from several cameras. We tested the method on 162 sequences of video data of human motions taken indoors and outdoors, and promising results were obtained. <s> BIB002 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> The quality of human silhouettes has a direct effect on gait recognition performance. This paper proposes a robust dynamic gait representation scheme, frame difference energy image (FDEI), to suppress the influence of silhouette incompleteness. A gait cycle is first divided into clusters. The average image of each cluster is denoised and becomes the dominant energy image (DEI). FDEI representation of a frame is constructed by adding the corresponding cluster's DEI and the positive portion of the frame difference between the former frame and the current frame. FDEI representation can preserve the kinetic and static information of each frame, even when the silhouettes are incomplete. This proposed representation scheme is tested on the CMU Mobo gait database with synthesized occlusions and the CASIA gait database (dataset B). The frieze and wavelet features are adopted and hidden Markov model (HMM) is employed for recognition. Experimental results show the superiority of FDEI representation over binary silhouettes and some other algorithms when occlusion or body portion lost appears in the gait sequences. <s> BIB003 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> Information fusion offers a promising solution to the development of a high-performance classification system. In this paper, the problem of multiple gait features fusion is explored with the framework of the factorial hidden Markov model (FHMM). The FHMM has a multiple-layer structure and provides an alternative to combine several gait features without concatenating them into a single augmented feature. Besides, the feature concatenation is used to directly concatenate the features and the parallel HMM (PHMM) is introduced as a decision-level fusion scheme, which employs traditional fusion rules to combine the recognition results at decision level. To evaluate the recognition performances, McNemar's test is employed to compare the FHMM feature-level fusion scheme with the feature concatenation and the PHMM decision-level fusion scheme. Statistical numerical experiments are carried out on the Carnegie Mellon University motion of body and the Institute of Automation of the Chinese Academy of Sciences gait databases. The experimental results demonstrate that the FHMM feature-level fusion scheme and the PHMM decision-level fusion scheme outperform feature concatenation. The FHMM feature-level fusion scheme tends to perform better than the PHMM decision-level fusion scheme when only a few gait cycles are available for recognition. <s> BIB004 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> Gait as a behavioural biometric is concerned with how people walk. However, most existing gait representations capture both motion and appearance information. They are thus sensitive to changes in various covariate conditions such as carrying and clothing. In this paper, a novel gait representation termed as Gait Entropy Image (GEnI) is proposed. Based on computing entropy, a GEnI encodes in a single image the randomness of pixel values in the silhouette images over a complete gait cycle. It thus captures mostly motion information and is robust to covariate condition changes that affect appearance. Extensive experiments on the USF HumanID dataset, CASIA dataset and the SOTON dataset have been carried out to demonstrate that the proposed gait representation outperforms existing methods, especially when there are significant appearance changes. Our experiments also show clear advantage of GEnI over the alternatives without the assumption on cooperative subjects, i.e. both the gallery and the probe sets consist of a mixture of gait sequences under different and unknown covariate conditions. (6 pages) <s> BIB005 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> In this paper, we propose a novel temporal template, called Chrono-Gait Image (CGI), to describe the spatio-temporal walking pattern for human identification by gait. The CGI temporal template encodes the temporal information among gait frames via color mapping to improve the recognition performance. Our method starts with the extraction of the contour in each gait image, followed by utilizing a color mapping function to encode each of gait contour images in the same gait sequence and compositing them to a single CGI. We also obtain the CGI-based real templates by generating CGI for each period of one gait sequence and utilize contour distortion to generate the CGI-based synthetic templates. In addition to independent recognition using either of individual templates, we combine the real and synthetic temporal templates for refining the performance of human recognition. Extensive experiments on the USF HumanID database indicate that compared with the recently published gait recognition approaches, our CGI-based approach attains better performance in gait recognition with considerable robustness to gait period detection. <s> BIB006 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> The strength of gait, compared to other biometrics, is that it does not require cooperative subjects. In previous work gait recognition approaches were evaluated using a gallery set consisting of gait sequences of people under similar covariate conditions (e.g. clothing, surface, carrying, and view conditions). This evaluation procedure, however, implies that the gait data are collected in a cooperative manner so that the covariate conditions are known a priori. In this work, gait recognition approaches are evaluated without the assumption on cooperative subjects, i.e. both the gallery and the probe sets consist of a mixture of gait sequences under different and unknown covariate conditions. The results indicate that the performance of the existing approaches would drop drastically under this more realistic experimental setup. We argue that selecting the most relevant gait features that are invariant to changes in gait covariate conditions is the key to develop a gait recognition system that works without subject cooperation. To this end, Gait Entropy Image (GEnI) is proposed to perform automatic feature selection on each pair of gallery and probe gait sequences. Moreover, an Adaptive Component and Discriminant Analysis (ACDA) is formulated which seamlessly integrates our feature selection method with subspace analysis for robust recognition, and importantly is computationally much more efficient compared to the conventional Component and Discriminant Analysis. Experiments are carried out on two comprehensive benchmarking databases: the CASIA database and the Southampton Human ID at a distance gait database (SOTON database). Our results demonstrate that the proposed approach significantly outperforms the existing techniques particularly when gait is captured with variable and unknown covariate conditions. <s> BIB007 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> In this paper, we proposed an improved two-level dynamic Bayesian network layered time series model (LTSM), which aims to solve the limitations hindering the application of available dynamic Bayesian networks, the hidden Markov model (HMM) and the dynamic texture (DT) model to gait recognition. In the first level, a gait silhouette or feature cycle is divided into several temporally adjacent clusters. Each cluster is modeled by a DT or logistic DT (LDT). In the second level, HMM is built to describe the relationship among the DTs/LDTs. Besides LTSM, LDT is also an improved dynamic Bayesian network presented in this paper to describe the binary image sequence, which introduces the logistic principle component analysis (PCA) to learning its parameters. We demonstrated the validity of LTSM with experiments on both the CMU Mobo gait database and CASIA gait database (dataset B), and that of LDT on the CMU Mobo gait database. Experimental results showed the superiority of the improved dynamic Bayesian networks. <s> BIB008 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> In this paper, we propose a novel gait representation-gait flow image (GFI) for use in gait recognition. This representation will further improve recognition rates. The basis of GFI is the binary silhouette sequence. GFI is generated by using an optical flow field without constructing any model. The performance of the proposed representation was evaluated and compared with the other representations, such as gait energy image (GEI), experimentally on the USF data set. The USF data set is a public data set in which the image sequences were captured outdoors. The experimental results show that the proposed representation is efficient for human identification. The average recognition rate of GFI is better than that of the other representations in direct matching and dimensional reduction approaches. In the direct matching approach, GFI achieved an average identification rate 42.83%, which is better than GEI by 3.75%. In the dimensional reduction approach, GFI achieved an average identification rate 43.08%, which is better than GEI by 1.5%. The experimental result showed that GFI is stronger in resisting the difference of the carrying condition compared with other gait representations. <s> BIB009 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> In gait recognition field, template-based approaches such as Gait Energy Image (GEI) and Chrono-Gait Image (CGI) can achieve good recognition performance with low computational cost. Meanwhile, CGI can preserve temporal information better than GEI. However, they pay less attention to the local shape features. To preserve temporal information and generate more abundant local shape features, we generate multiple HOG templates by extracting Histogram of Oriented Gradients (HOG) of GEI and CGI templates. Experiments show that compared with several published approaches, our proposed multiple HOG templates achieve better performance for gait recognition. <s> BIB010 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> Gait Energy Image (GEI) is an efficient template for human identification by gait. However, such a template loses temporal information in a gait sequence, which is critical to the performance of gait recognition. To address this issue, we develop a novel temporal template, named Chrono-Gait Image (CGI), in this paper. The proposed CGI template first extracts the contour in each gait frame, followed by encoding each of the gait contour images in the same gait sequence with a multichannel mapping function and compositing them to a single CGI. To make the templates robust to a complex surrounding environment, we also propose CGI-based real and synthetic temporal information preserving templates by using different gait periods and contour distortion techniques. Extensive experiments on three benchmark gait databases indicate that, compared with the recently published gait recognition approaches, our CGI-based temporal information preserving approach achieves competitive performance in gait recognition with robustness and efficiency. <s> BIB011 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Introduction Approach <s> A method combined the modified Lucas-Kanade Gait Flow Image(LK-GFI) with the view was proposed to solve the problem that personal identification based on a gait is sensitive to view change.The Lacus-Kanade optical method was used to compute the optical flow between two silhouettes to construct LK-GFI,and the view was obtained according to the walking direction of the person.The LK-GFI database for the target at different views was established,then the new person's view and LK-GFI were extracted.At last,the similarity between the new person's LK-GFI and the target's LK-GFI at the same view was computed by the Euclidian distance method.The performance of this method was evaluated on the data in the CASIA database and the data obtained in indoor lab environment,and the False Rejection Rate(FRR) is 7.95% and 9.12% respectively.It is reduced by 12.5% and 14.45% respectively compared with that of the Gait Energy Image(GEI),and by 7.77% and 6.74% respectively compared with that of the Gait Flow Image(GFI).The proposed method has high recognition accuracy,strong real-time and the robustness to view changes. <s> BIB012
Gait information accumulation approach only reformulates a video sensor-based gait sequence by using templates as holistic feature and arguably loses some intrinsic dynamic characteristics of the gait pattern. In order to weaken this effect, gait information introduction approach introduces dynamic information to the static silhouette images based on GEI by adopting the mathematical transformation methods of average, difference and motion regions extraction, etc. The general gait information introduction approaches include: Motion Information Energy Image (MIEI) BIB002 , Frame Difference Energy Image (FDEI) BIB003 BIB004 BIB008 , Enhance Gait Energy Image (EGEI)   BIB001 , Chrono Gait Image (CGI) BIB006 BIB010 BIB011 , Gait Flow Image (GFI) BIB009 BIB012 , and Gait Entropy Image (GenI) BIB007 BIB005 , etc. Masoud et al. BIB002 applied the weighted average idea to GEI, and Motion Information Energy Image (MIEI) was devised. MIEI was started with a weighted average at time (frame) n . MIEI is computed as follows: where α is a parameter within [0,1] . ( , , ) E x y n can be updated consecutively. The recent image makes a greater contribution to ( , , ) E x y n . More dynamic discriminative information is introduced to GEI by selecting appropriate ( , , 1) E x y n − and α . However, the values of ( , , 1) E x y n − and α vary from To suppress the influence of silhouette incompleteness for identification, Chen et al. BIB003 proposed Frame Difference Energy Image (FDEI). A gait cycle is divided into clusters and the dominant energy image (DEI) is obtained by denoising the averaged image of each cluster. The frame difference is calculated by subtracting two consecutive frames. FDEI representation is constructed as the summation of its corresponding cluster's DEI and the positive portion of its frame difference. FDEI is defined as follows: if B x y n B x y n F x y n B x y n B x y n otherwise where C N is the number of frames in the th C cluster, and it represents the time (frame) set of silhouettes. Average distortion is used to choose the cluster number BIB003 , which decreases with the increased cluster number. If the average distortion does not change appreciably beyond certain number, it can be chosen as the cluster number. When the value of cluster number is very small, some useful information for identification may miss. The larger cluster number leads to great computation complexity, but has little improvement on the recognition performance. The threshold T varies with different periods or subjects, depending on the quality of the silhouettes. ( , ,0) B x y is viewed as the last frame in a period. When ( , , ) B x y n is incomplete and ( , , 1) B x y n − is complete, the missing portions of the frame are contained in ( , , ) F x y n . When the ( , , ) B x y n and ( , , 1) B x y n − are both incomplete, the missing portions can be compensated by ( , ) c DEI E x y . FDEI representation helps to suppress the influence of the missing portions and preserve the characteristics of ( , , ) B x y n . FDEI is robust to incomplete silhouette images. Moreover, FDEI embodies both static and kinetic information between frames. Figure 5d -i demonstrate some images during the construction of FDEI, where Figure 5d ,e show silhouettes ( , , 1) B x y n − and ( , , ) B x y n , respectively. Figure 5f shows the movement portion of ( , , ) B x y n . A GEI of the cluster is shown in Figure 5g . Figure 5h embodies the dominant energy of GEI, which is obtained by denoising Figure 5g . Figure 5i is FDEI of ( , , ) B x y n . Comparing (i) with Figure 5d , it can be seen that FDEI contains the movement portion and partially compensates the incompleteness of ( , , ) B x y n . Yang et al.  also devised an Enhance Gait Energy Image (EGEI) representation method. The method applied dynamic region analysis to improve dynamic information of GEI. Then a better performance can be attained than the conventional GEI method. The intensity in GEI indirectly reflects the time spent at each stance: the regions with high intensity and low intensity marked with "I" and where A is the total number of classes in the training set. N is the number of samples in each class. The total number of the samples in the training set is M NA = . ( , ) Temporal information of GEI may be lost, whereas it reduces the effect of the noises. In order to well preserve the temporal information of gait patterns, Wang et al. BIB006 put forward a multi-channel temporal encoding technique, named as Chrono Gait Image (CGI), to encode a gait sequence to a multi-channel image. CGI is defined as follows:  where p is the number of silhouettes in the 1/4 gait period. period which the th t frame belongs to. Figure 7 represents the process of generating CGI. However, CGI may lose some dynamic information, such as the frequency information. The accuracy of CGI depends on the number of 1/4 gait period in a silhouette sequence. Optical flow is a mode of target motion in scenes. Optical flow has been well used in the field of moving target detection, segmentation and tracking. Toby et al. BIB009 introduced optical flow to gait representation, and presented Gait Flow Image (GFI), which is obtained as in Equation (15). GEI is sensitive to static condition variation, such as clothing, carrying condition (backpack, briefcase, handbag, etc.), and shoe-wear. Bashir et al. BIB007 proposed the Gait Entropy Image (GEnI) to distinguish the dynamic and static areas of GEI by measuring Shannon entropy at each pixel location in GEI. The intensity value of the silhouettes at a fixed pixel location is considered as a discrete random variable. Shannon entropy measures the uncertainty associated with the random variable over a complete gait cycle. GEnI is computed as: 2 2 ( , ) ( , )log ( , ) (1 ( , ))log (1 ( , )) GEnI measured the relevance of gait features extracted from GEI, and automatically selected static condition invariant features for gait recognition. Since the dynamic region has more uncertainty, the intensity values of GEnI are larger in the dynamic region and smaller in the static region. Figure 9 shows some GEI and GEnI samples of one person walking normally, walking with bag and walking while wearing a coat, where (a, c, e) are the GEI samples, and (b, d, f) are the corresponding GEnI ones, respectively. Dynamic and static information could be easily distinguished by constructing the Shannon entropy in GEI. (1 ) , (0,1)  (1 ( , )) log (1 ( , )) We make √ and × represent whether the Class Energy Image with or without the type of the motion information, respectively. O denotes computational complexity. Suppose the size of the silhouette is m n × , η is the numbers of silhouettes in a gait cycle. Gait information introduction approach can highlight the dynamic information, meanwhile preserving the static information. Table 2 characterizes the expressions, motion information and computational complexity of the gait information introduction approach. This kind of approach is insensitive to noises, and not critical to the identification method. In addition, the computational complexity of Class Energy Images in this section has significant differences. Especially, it is obvious that CGI and GFI have extremely higher computational complexity than others.