Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> A view-based approach to the representation and recognition of human movement is presented. The basis of the representation is a temporal template-a static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: The first value is a binary value indicating the presence of motion and the second value is a function of the recency of motion in a sequence. We then develop a recognition method matching temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on standard platforms. <s> BIB001 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> Human motion analysis is a recent topic of interest among the computer vision and video processing community. Research in this area is motivated by its wide range of applications such as surveillance and monitoring systems. In this paper we describe a system for recognition of various human actions from compressed video based on motion history information. We introduce the notion of quantifying the motion involved, through what we call Motion Flow History (MFH). The encoded motion information readily available in the compressed MPEG stream is used to construct the coarse Motion History Image (MHI) and the corresponding MFH. The features extracted from the static MHI and MFH compactly characterize the spatio-temporal and motion vector information of the action. Since the features are extracted from the partially decoded sparse motion data, the computational load is minimized to a great extent. The extracted features are used to train the KNN, Neural network, SVM and the Bayes classifiers for recognizing a set of seven human actions. The performance of each feature set with respect to various classifiers are analyzed. q 2003 Elsevier B.V. All rights reserved. <s> BIB002 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> This paper presents a novel approach for human recognition by combining statistical gait features from real and synthetic templates. Real templates are directly computed from training silhouette sequences, while synthetic templates are generated from training sequences by simulating silhouette distortion. A statistical feature extraction approach is used for learning effective features from real and synthetic templates. Features learned from real templates characterize human walking properties provided in training sequences, and features learned from synthetic templates predict gait properties under other conditions. A feature fusion strategy is therefore applied at the decision level to improve recognition performance. We apply the proposed approach to USF HumanID Database. Experimental results demonstrate that the proposed fusion approach not only achieves better performance than individual approaches, but also provides large improvement in performance with respect to the baseline algorithm. <s> BIB003 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> We present a robust representation for gait recognition that is compact, easy to construct, and affords efficient matching. Instead of a time series based representation comprising of a sequence of raw silhouette frames or of features extracted therein, as has been the practice, we simply align and average the silhouettes over one gait cycle. We then base recognition on the Euclidean distance between these averaged silhouette representations. We show, using the recently formulated gait challenge problem (www.gaitchallenge.org), that the improvement in execution time is 30 times while possessing recognition power that is comparable to the gait baseline algorithm, which is becoming the comparison standard in gait recognition. Experiments with portions of the average silhouette representation show that recognition power is not entirely derived from upper body shape, rather the dynamics of the legs also contribute equally to recognition. However, this study does raise intriguing doubts about the need for accurate shape and dynamics representations for gait recognition. <s> BIB004 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> Recently, gait recognition for human identification has received substantial attention from biometrics researchers. Compared with other biometrics, it is more difficult to disguise. In addition, gait can be captured in a distance by using low-resolution capturing devices. In this paper, we proposed a new representation for human gait recognition which is called Motion Silhouettes Image (MSI). MSI is a grey-level image which embeds the critical spatio-temporal information. Experiments showed that MSI has a high discriminative power for gait recognition. The recognition rate is around 87% in SOTON dataset by using MSI for recognition. The recognition rate is quite promising. In addition, MSI can also reduce the storage size of the dataset. After using MSI, the storage size of SOTON has reduced to 4.2MB. <s> BIB005 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> In this paper, we propose a new spatio-temporal gait representation, called Gait Energy Image (GEI), to characterize human walking properties for individual recognition by gait. To address the problem of the lack of training templates, we also propose a novel approach for human recognition by combining statistical gait features from real and synthetic templates. We directly compute the real templates from training silhouette sequences, while we generate the synthetic templates from training sequences by simulating silhouette distortion. We use a statistical approach for learning effective features from real and synthetic templates. We compare the proposed GEI-based gait recognition approach with other gait recognition approaches on USF HumanID Database. Experimental results show that the proposed GEI is an effective and efficient gait representation for individual recognition, and the proposed approach achieves highly competitive performance with respect to the published gait recognition approaches <s> BIB006 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> In this paper, an improved temporal template called gait history image (GHI) is proposed for human gait representation and recognition. Comparing with other temporal template methods, GHI models gait more comprehensively: Static and dynamic characteristics, as well as spatial and temporal variations, can be represented. The time duration of the GHI template is controlled by a finer period resolution (1/4 of a gait cycle). We use statistical approach to learn the discriminating features from GHIs, and gait recognition experiments were performed on the CASIA and USF gait databases. The methods using GHI template presented better recognition performances than the baseline algorithm and gait energy image (GEI) method. <s> BIB007 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> This paper utilizes the periodicity of swing distances to estimate gait period. It shows good adaptability to low quality silhouette images. Gait moment image (GMI) is implemented based on the estimated gait period. GMI is the gait probability image at each key moment in gait period. It reduces the noise of the silhouettes extracted from low quality videos by gait probability distribution at each key moment. Moment deviation image (MDI) is generated by using silhouette images and GMIs. As a good complement of gait energy image (GEI), MDI provides more motion features than the basic GEI. MDI is utilized together with GEI to represent a subject. The nearest neighbor classifier is adopted to recognize subjects. The proposed algorithm is evaluated on the USF gait database, and the performance is compared with the baseline algorithm and two other algorithms. Experimental results show that this algorithm achieves a higher total recognition rate than the other algorithms. <s> BIB008 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> A moment gait energy image(MGEI)based gait recognition algorithm is presented.Gait period is estimated from swing distances,and key moments of a gait cycle are specified.The mean of all the silhouette images at a key moment is called the mean moment image.The MGEI at each key moment is calculated from all the mean moment images in gait period.It improves sil- houette's quality by gait probability distribution at each key moment,and provides more motion features than the basic gait energy image(GEI).Then a method of gait feature extraction is proposed based on MGEI.The deviations of silhouette images from MGEI at every key moment are cumulated into an image,which is utilized together with GEI to represent a subject.The nearest neighbor classifier is adopted to recognize subjects.The proposed algorithm is evaluated on USF dataset,and the performance is compared with the baseline algorithm and two other new algorithms.Experimental result shows that this algorithm achieves higher overall recognition rate than the other algorithms. <s> BIB009 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> Traditional image representations are not suited to conventional classification methods such as the linear discriminant analysis (LDA) because of the undersample problem (USP): the dimensionality of the feature space is much higher than the number of training samples. Motivated by the successes of the two-dimensional LDA (2DLDA) for face recognition, we develop a general tensor discriminant analysis (GTDA) as a preprocessing step for LDA. The benefits of GTDA, compared with existing preprocessing methods such as the principal components analysis (PCA) and 2DLDA, include the following: 1) the USP is reduced in subsequent classification by, for example, LDA, 2) the discriminative information in the training tensors is preserved, and 3) GTDA provides stable recognition rates because the alternating projection optimization algorithm to obtain a solution of GTDA converges, whereas that of 2DLDA does not. We use human gait recognition to validate the proposed GTDA. The averaged gait images are utilized for gait representation. Given the popularity of Gabor-function-based image decompositions for image understanding and object recognition, we develop three different Gabor-function-based image representations: 1) GaborD is the sum of Gabor filter responses over directions, 2) GaborS is the sum of Gabor filter responses over scales, and 3) GaborSD is the sum of Gabor filter responses over scales and directions. The GaborD, GaborS, and GaborSD representations are applied to the problem of recognizing people from their averaged gait images. A large number of experiments were carried out to evaluate the effectiveness (recognition rate) of gait recognition based on first obtaining a Gabor, GaborD, GaborS, or GaborSD image representation, then using GDTA to extract features and, finally, using LDA for classification. The proposed methods achieved good performance for gait recognition based on image sequences from the University of South Florida (USF) HumanID Database. Experimental comparisons are made with nine state-of-the-art classification methods in gait recognition. <s> BIB010 </s> Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review <s> Gait Information Accumulation Approach <s> A method based on sparse representation for gait recognition was proposed using the CASIA-B and CUSD database.The gait silhouette was centralized and normalized first,then the AEI(active energy image) was calculated based on the previous operation and used as the feature image for gait recognition.Two methods were used to establish the dictionary and calculate the decomposition coefficients for the sparse representation: one for reconstruction,and the other for discrimination.OMP(orthogonal matching pursuit) algorithm was used for coefficients decomposition.The result of experiment shows that the proposed method can effectively recognize the gait,and the accuracy of the recognition is high and recognizing speed is much faster. <s> BIB011
The gait information accumulation approach makes an original video sensor-based gait silhouette sequence represent as one or several matrix-like second-order images by using mathematical methods of average, difference, maximum and minimum operation, etc. The gait information accumulation approach is insensitive to incidental silhouette errors, and performs better and provides richer information than the original binary gait image. The common gait information accumulation approaches include Motion Energy Image (MEI) BIB001 , Motion History Image (MHI) BIB001 BIB002 , Motion Silhouettes Image (MSI) BIB005 , Gait Energy Image (GEI) BIB006 BIB003 , Averaged Silhouette (AS) BIB004 , Gait History Image (GHI) BIB007 , forward Single-step History Image (fSHI)  , backward Single-step History Image (bSHI)  , Active Energy Image (AEI)  BIB011  , Gait Moment Image (GMI) BIB008 , Moment Gait Energy Image (MGEI) and Gait Deviation Image (GDI) BIB009 , etc. In  where n is the frame number (moment of time) of a silhouette sequence, x and y are values in the 2D image coordinate. τ is the duration of a current silhouette in the sequence. The pixel value of ( , , ) B x y n ranges within [0,1] . ( , , ) D x y n is the binary difference silhouette image, which indicates regions of motion. For example, ( , , ) 1 D x y n = represents a motion occurrence in the th n frame (time) on the coordinate point ( , ) x y . MEI accumulates all regions of motion in a gait sequence. MHI is a gray image with temporal information. The value of MHI is associated with the current moment. MEI and MHI have less computational complexity, but the static information is not sufficient. Examples of MEI and MHI are shown in Figure 3a ,b. The two templates are the basis of the later research on the behaviors and gait recognition. Lam et al. BIB005 employed Motion Silhouettes Image (MSI) to characterize a motion image sequence. MSI is generated in nearly the same way as MHI. Pixel intensity of MSI is a function of the temporal history of motion at this point, and MHI is generated by using the following algorithm: MSI, which is simpler than MHI, is a gray image with temporal information. The pixel value shows the motion history at this pixel. Figure 3c shows an example of MSI. Gait identification is a special case of behavior recognition. In 2004, Han et al. BIB003 proposed Gait Energy Image (GEI). The grey-level GEI can be created from averaging the silhouettes with Equation (3). where N denotes the number of the binary silhouette images in a gait cycle. GEI reflects temporal length of each posture in a complete gait period. A pixel with higher intensity value in GEI means that human walking occurs more frequently at this position. GEI preserves the dynamic and static (shape) information of a gait sequence. The common static information is the proportion of the human body, clothing and bags, etc. Furthermore, there is no consideration of the time that normalizes each silhouette. Figure 3d presents an example of MSI. Liu et al. BIB004 used average silhouette to characterize gait features. Suppose a sequence of silhouettes, { (1), , ( )} S S S M =  , the th i average silhouette (AS) in a gait cycle was expressed as Gait N denotes the numbers of silhouettes in a gait cycle. It is noteworthy that the theory of AS and GEI is similar. Inspired by MHI, Liu et al. BIB007 adopted Gait History Image (GHI) to characterize the motion image sequence. GHI is obtained as follows: where P is the number of the frames in a quarter cycle of a silhouette image sequence, and GHI is generated from 1/4 gait period. The pixel values in GHI represent the temporal changes. GHI not only contains the dynamic and static information, but also inherits the characteristics of MHI which can reflect temporal variation. However, GHI computed from a quarter gait period, it will lose amount of useful information. Figure 3e shows a GHI sample of one person. Based on the idea of MHI in behavior recognition, Chen et al.  proposed a Single-step History Image (SHI). The forward and backward difference image between two adjacent silhouettes can obtain forward Single-step History Image (fSHI) and backward Single-step History Image (bSHI), respectively. fSHI and bSHI are defined as follows: where N λ is the number of the frames in the th GEI is easily influenced by clothing and carrying conditions, meanwhile it neglects some dynamic information. In order to address these problems, Zhang et al.  presented Active Energy Image (AEI), AEI is obtained as follows:  AEI aims at extracting the active regions by calculating the difference between two adjacent silhouettes in the gait sequence. AEI contains more temporal characteristics for discriminant than GEI. Moreover, AEI can reduce the influence of carrying conditions. When clothing and loading change little between adjacent silhouettes, the influence could be ignored. However, AEI only makes use of the dynamic parts in the silhouette images without consideration of the static information. Figure 3h -j shows some AEI samples of a subject under normal walking, walking with bag and walking on coat conditions. Each cycle only includes one GEI or GHI template, which easily leads to the problem of insufficient training samples. To address this problem, Ma et al. BIB008 utilized Gait Moment Image (GMI) to express a silhouette sequence. GMI is the gait probability image at each key moment in all gait cycles. The corresponding gait images at a key moment are averaged as GEI of this key moment. GMI at the th k key moment is calculated as: where C is the number of the gait cycles in a gait sequence. Figure 4a shows five GMIs of the same person. Ma et al. BIB009 further improved the image quality of each key frame, and uniformly choose S interested moments as the key frames in a gait cycle. Moment Gait Energy Image (MGEI) at the th k key moment can be given as follows: where 1 r is the decline coefficient. Smaller values of 1 r would make MGEI quite similar to ( , , ) B x y k . While bigger values of BIB010 r would make each frame lose uniqueness. Some key frames are sufficiently employed to create the MGEI, and the way of its calculation is greatly different from GEI. Furthermore, MGEI has temporal information between frames. Some examples of MGEI are shown in Figure 4c . Furthermore, Gait Deviation Image (GDI), which represented a kind of accumulation of the deviations between original silhouette images and the moment probability images, is obtained as follows: where N is the gait period in a gait sequence.