Spatial Temporal Graph Attention Network for Skeleton-Based Action Recognition

It's common for current methods in skeleton-based action recognition to mainly consider capturing long-term temporal dependencies as skeleton sequences are typically long (>128 frames), which forms a challenging problem for previous approaches. In such conditions, short-term dependencies are few formally considered, which are critical for classifying similar actions. Most current approaches are consisted of interleaving spatial-only modules and temporal-only modules, where direct information flow among joints in adjacent frames are hindered, thus inferior to capture short-term motion and distinguish similar action pairs. To handle this limitation, we propose a general framework, coined as STGAT, to model cross-spacetime information flow. It equips the spatial-only modules with spatial-temporal modeling for regional perception. While STGAT is theoretically effective for spatial-temporal modeling, we propose three simple modules to reduce local spatial-temporal feature redundancy and further release the potential of STGAT, which (1) narrow the scope of self-attention mechanism, (2) dynamically weight joints along temporal dimension, and (3) separate subtle motion from static features, respectively. As a robust feature extractor, STGAT generalizes better upon classifying similar actions than previous methods, witnessed by both qualitative and quantitative results. STGAT achieves state-of-the-art performance on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400. Code is released.


Introduction
As a special issue of action recognition, skeleton-based action recognition has attracted much attention recently due to the fast development of pose estimation algorithms [2,9] and wearable motion capture equipment. It involves predicting an action label from a sequence of 2D or 3D skeleton representations. Compared to action recognition in videos [40,39,3], skeleton data is more compact in 2D [14,41] or 3D [23,27] format and suffers less from environmental factors (e.g. camera motion, light transformations and viewpoint changing), which makes it a more robust representation of human action sequences.
It's usual for current methods to mainly tackle long-term temporal dependencies as skeleton sequences are rather long (>128 frames), which forms a serious obstacle for previous methods to model human dynamics through a whole video.  However, in such conditions, critical short-term temporal dependencies are few formally considered. In contrast with long-term dependencies which recognize action sequences from an overall perspective, short-term dependencies distinguish similar sequences on small temporal scales with important details. For example, 'Eating' and 'Tooth brushing' have quite similar movements in both upper and lower body where hands are raised next to mouth for a long while, holding food or brushing teeth, which is quite similar in long sequences. To classify them relies on whether there exists subtle movement of the hand to brush horizontally, which is short-term and rather critical. Few methods [25,43] have realized the importance of short-term motion and proposed dilated temporal convolutions to capture short-term temporal movements. We propose a general framework to perform spatial-temporal modeling, achieving superior performance than them under similar computational budgets as shown in Sec. 4.3. Most existing methods employ interleaving spatial-only and temporal-only modules to extract spatial features and capture long-term temporal dependencies, respectively. A typical approach is to first use a graph module to extract spatial representations at each time step, and then deploy recurrent or temporal convolution layers to build temporal relationships ( Fig.1(a)) [17,32,20,30,41]. Such design are efficient and effective for long-term modeling, but hinders direct information flow in local spatial-temporal neighborhoods. For example, messages have to pass through a spatial module and a temporal module to reach nodes in adjacent frames, which are inevitably weakened in multiple transitions. Especially, large temporal kernels adopted after spatial modules get more information mixed and may cause subtle motion impaired, where short-term temporal dependencies may be ineffectively captured by such factorized modeling.
In this work, we propose STGAT with spatial-temporal modeling to specially capture short-term dependencies. We retain the temporal-only modules for longterm modeling and give spatial-only modules more freedom to perform local spatial-temporal modeling. It builds local spatial-temporal graphs by connecting nodes in local spatial-temporal neighborhoods and dynamically constructing their relationships (Fig.1(c)). Thus messages can directly reach other joints in lo-cal neighborhoods without multiple transmission costs. The connection strength of edges is adaptively determined to better fit local movement patterns. For further releasing the potential of STGAT, we propose several light-weight modules to reduce local feature redundancy. These modules make effect in narrowing the scope of self-attention mechanism, dynamically weighting joints along temporal dimension and separating subtle motion from static features. Equipped with the ability in spatial-temporal modeling, STGAT exhibits great progress on classifying similar action pairs, witnessed by both qualitative and quantitative results. As a result, STGAT achieves state-of-the-art performance on three large-scale datasets: NTU RGB+D 60 [27], NTU RGB+D 120 [23], and Kinetics Skeleton 400 [14]. Code is releasedhttps://github.com/hulianyuyy/STGAT.

Skeleton-Based Action Recognition
Earlier methods always make use of hand-crafted features [37,38] to depict human dynamic representations. However, they fail to model dependencies across spacetime and lack enough ability in extracting high-dimensional features. Later, convolutional neural network(CNNs) and recurrent neural networks(RNNs) become mainstream by modeling human dynamics as a pseudo-image [15,18,19] or a series of coordinates along time [8,24,21,44,32]. Nevertheless, they overlook the internal relationships between joints which are better captured by graph networks due to their natural advantage over handling non-euclidean data. Graph networks in skeleton-based action recognition fall into two streams: spatial perspective and spectral perspective. ST-GCN [41] belonging to the spatial perspective firstly models human structure as a spatial graph to construct node relationships. Later methods mainly make efforts in enlarging the receptive field [10,20,30,4], combining another stream [30,20,25,29], dividing graphs into structural ones [20,34], design network structure [13,26,33,6,4], combining adaptive learning [30,42,29,45,12,31,1].

Self-Attention Mechanism with Skeleton Data
One of the main benefits of self-attention mechanism is to adaptively adjust relationships with neighbors according to their responses. A self-attention module flexibly computes the response of a position with others' in an embedding space and then aggregates their weighted features. Self-attention mechanism has proven its power in many fields [5,22,40,36] and is recently introduced to model relationships [42,29,30,25,45,31] in skeleton-based action recognition. 2s-AGCN adds an adaptively computed graph with the physically predefined graph to flexibly model joint relationships, while MS-G3D [25], CA-GCN [46] and dynamic-GCN [42] inherit it. DSTA-Net [31] and SGN [45] proposes to alternatively employ fully attention-based spatial modeling and temporal modeling for capturing movement patterns. In contrast with previous approaches, our method doesn't rely on a certain self-attention strategy, but proposes a spatial-temporal modeling framework for capturing long-term and especially short-term dependencies.

Preliminaries
Notations. Formally, a graph is always represented as G=(V, E ), where V = {v 1 ,...,v N } is a set of N graph nodes representing joints and E is a series of graph edges representing bones between joints. An adjacent matrix A with size N ×N is adopted to depict bones where A i,j represents the connection strength between node i and j, which equals zero if they have no connections. As for input, action sequences have a node features set represented as a feature tensor X C×T ×N where each node v n ∈ N has a C dimensional feature vector over total T frames.
Graph Convolutional Networks. Given the input tensor X and graph structure A, layer-wise graph convolution at each time step can be implemented as: where X out and X in separately corresponds to the output and input features and W is a trainable weight matrix. Λ is the diagonal degree matrix of A to employ normalization and I adds self-loops to keep identity features. At last, σ(·) acts as an activation function for output.

Spatial-Temporal Graph Attention Network
Following the statement in Sec. 1, short-term dependencies are critical cues for classifying similar action pairs. We propose a general framework to capture shortterm dependencies by performing spatial-temporal modeling, detailed as follows.
Building Spatial-Temporal Graphs. To better capture short-term dependencies, in this paper we give spatial-only modules more freedom for shortterm aggregation, while maintaining the role of temporal-only modules for longterm modeling. We extend spatial-only modules to build a local robust spatialtemporal feature aggregator which directly exchange messages between nodes in a local spatial-temporal neighborhood. We start by building a spatial-temporal graph consisting of nodes in a local cross-spacetime neighborhood, which have links with not only other nodes in their original spatial graph but also their neighbors in neighboring timestamps. In this way, each node can directly aggregate messages from other local spatial-temporal neighbors. Let us consider a sliding window of size τ and dilation d at each time step over input sequences, which generates a local action sequence at each time step t represented as X t τ = {x t−τ /2:t+τ /2 ∈ R C×τ N |t ∈ Z, 0 ≤ t < T }. Here, τ controls the local sequence length along time while d represents the dilation rate by picking a frame every d frames. Dilated window allows us to control the interval between sampled frames to fit local movement patterns. Then, for each frame x t , a unique regionally spatial-temporal adjacent matrix A t τ is organized by enumerating all possible neighbors in local spatial-temporal neighborhood X t τ as: where A t (τ ) represents a cross-time graph with size N × N by establishing relationships between nodes at current timestamp t with their neighbors at τ neighboring frames in a local spatial-temporal neighborhood. If we take one node of current timestamp t as an example (i.e. one column of A t τ ), it's connected to τ × N neighbors consisting of τ frames with N nodes at each timestamp. By incorporating A t τ into graph operations, the output of STGAT for each frame x t can be obtained as: Adaptive Computation for Spatial-Temporal Graph. For well distinguishing similar action pairs, more attention should be paid to aggregate messages from important joints and less to other joints. We incorporate self-attention mechanism to adaptively compute edge weights in local spatial-temporal graphs to achieve this goal. Following the widely used self-attention formulation, a generic self-attention operation in graph networks at each time step can be defined as: where i is the index of an output position, j is the index that enumerates all possible neighbors and A i is the computed edge weights of node v i with others.
Here, the pairwise function f (·) computes the affinity of features between node v i and v j . g(·) offers an embedding feature for node v j . C(·) adds normalization for the result. Given the adjacent matrix A, a weighted output X out can be computed as: A more stable and flexible choice is employing multi-head attention modules to learn various kinds of edge weights. Specially, S independent self-attention modules are employed to learn different graph structures, as: Here A s and W s are the computed adjacent matrix and weight matrix separately of s th head. By incorporating our spatial-temporal graph A t τ and our sampled local action sequence X t τ in Eq. 5, we arrive at the final form to compute edge weights and aggregate messages from neighbors in spatial-temporal graphs: where A t τ,s represents a unique computed spatial-temporal graph at timestamp t with temporal length τ of the s th head. X out is the accumulated output of all heads with different spatial-temporal graphs.
Based on above design, our STGAT is a general framework which doesn't rely on a certain self-attention strategy and can be built with self-attention operators or any attention-based approaches by inflating their spatial-only modules to construct cross-spacetime edges for short-term modeling.
Discussion. We give some in-depth analysis for STGAT as follows: (1) Compared with MS-G3D [25] which builds a spatial-temporal graph with size τ N × τ N for cross-spacetime modeling, ours owns size τ N × N . Thus, MS-G3D introduces more redundant messages from neighbors and cost N times of computations than ours. More importantly, MS-G3D assigns same edge weights for joints in a local spatial-temporal neighborhood ( fig.1(b)), which fails to distinguish beneficial messages and allows too much information to flood into current node without filter. In contrast, we dynamically construct joint relationships and thus is able to aggregate representative features. In this condition, MS-G3D deploys two pathways and proposes multiscale temporal convolutions to capture various short-term motion but still performs worse than ours. 2. STGAT is analogous to 3DConv [35,3] in modeling spatial-temporal dependencies by a weighted aggregation of regional cross-spacetime messages. While the parameters of 3D convolution kernels are structurally learned and fixed for samples, STGAT adaptively computes the edge weights for each sample and is less prone to overfit than 3DConv during training.

Reducing local spatial-temporal redundancy
While STGAT is theoretically effective for spatial-temporal aggregation, we empirically found direct aggregation of messages from spatial-temporal neighbors brings redundant static information, resulted from similar contents in adjacent frames. We propose several simple modules to reduce local spatial-temporal redundancy and further release the potential of STGAT, resulting in a robust feature extractor.
Separate Learning. Compared to the original N ×N spatial graph, STGAT builds a τ N ×N spatial-temporal graph to model local dependencies across spacetime. Obviously, the size of adjacent matrix increases τ times. That makes it harder for self-attention operators to adaptively construct joint relationships among neighbors with similar contexts. As we found in practice, as node number grows, attention-based modules tend to assign uniform weights around 1/(τ ×N ) for each joint, which thus gradually loses the ability to adaptively construct relationships. We propose Separate Learning Strategy to limit node numbers and keep beneficial information lossless to relieve this problem. Inspired by groupwise convolution [16], we propose to divide joints in a local spatial-temporal region into several small groups, where STGAT only weights edges within groups.

Motion features Motion features
Explicit motion modeling Normal Fig. 2. Comparison of our proposed Explicit Motion Modeling strategy with normal 1 × 1 implementation. We try to separate short-term motion in local spatial-temporal region from static features for better short-term modeling. Thus, the scope of self-attention operators is narrowed and the computation costs are proportionally lowered down as well. Features from different groups are concatenated as outputs which won't bring information loss. Practically, while there exist lots of methods to group joints, we choose to divide them according to their distances with target joint.
Dynamic Temporal Weighting. As defined in Eq. 2 and 3, STGAT builds spatial-temporal graphs to model cross-spacetime information. We notice that some keyframes play more important roles than others where considering frames equally may hurt modeling short-term motion. We propose two methods to dynamically weight frames regionally as follows and compare them in the experiments: (1) We add a learnable parameter T DT W with size S ×τ to weight frames for S heads with temporal duration τ , which is updated through network propagation. Note that T DT W structurally considers the weights of frames in each head. It's fixed for each sample during inference.
(2) Inspired by SENet [11], we design an adaptive temporal weighting module to weight frames by considering the short-term motion in a local spatial-temporal region. We first perform 1-hop graph convolutions for current frame X t to aggregate local spatial features. Then, a 1 × 1 convolution kernel is utilized to enhance features of current frame X t . We subtract features of regional neighbors from those of current frame to obtain motion features, which reflect the movement degree of neighboring frames against current one. Finally, after pooling along channel and temporal dimension, they are passed through a sigmoid function to weight frames in local spatial-temporal region. These weights are computed adaptively in accordance with input for regional weighting. An Illustration can be found in the supplementary material.  Explicit Motion Modeling. Considering neighboring frames always contain similar contents, we infer only one frame can provide enough static information and short-term motion benefits recognition more, for reducing local feature redundancy. We propose Explicit Motion Modeling strategy to separate shortterm motion in local spatial-temporal neighborhood from static features. As illustrated in fig.2, compared to original 1 × 1 transformations, we first split the features along temporal dimension. And then, we subtract features of adjacent frames from those of current frame to obtain motion features. In this condition, only static features of current timestamp is reserved and short-term motion is obtained. With appropriate padding, these motion features pass a 1 × 1 convolution kernel for enhancement and are finally concatenated with current frame's features as output. Compared with normal 1 × 1 convolution, this design introduces no additional parameters but enhances the ability for distinguishing short-term motion.

···
Discussion. These three modules are lightweight and can be painlessly incorporated into STGAT, with quite little costs. Only Dynamic Temporal Weighting consumes S ×τ parameters, which are negligible in deep-learning-based networks. They make powerful effect for reducing local redundancy, resulting in superior performance of STGAT, as witnessed in sec. 4.3.

Model Architecture
Overall Architecture. Our method is a general framework which can be flexibly constructed upon self-attention operators or previous attention-based approaches. To demonstrate the effectiveness of STGAT, we choose DSTA-Net [31] as our baseline, which employs fully attention-based spatial-only modules and temporal-only modules alternatively. It's already among the top-performing methods currently. As illustrated in fig.3, our model is consisted of r spatial-temporal graph attention blocks (STGABs) followed by a series of global average pooling, fully connected and softmax operations to perform prediction. Each STGAB contains a spatial-temporal graph attention module to capture regional crossspacetime dependencies and a temporal convolution module to perform longrange temporal information aggregation. For simplicity, BN and ReLU and neglected in the figure.
Multiple-Stream Fusion. Inspired by 2s-AGCN [30], beneficial bone information is modeled by another stream. Following our baseline, we also calculate motion for both joint and bone streams to obtain another two motion streams. The softmax scores of input streams are averaged to give the final prediction.

Datasets
NTU RGB+D 60 and NTU RGB+D 120. NTU RGB+D 60 [27] and NTU RGB+D 120 [23] are both large-scale widely used action recognition datasets which separately contain 56578 action sequences over 60 classes and 113945 action sequences from 120 classes. All samples are performed by 40 or 106 different volunteers modeled by their 3D locations with N =25. Two experiment setups are recommended by the author: (1) Cross-subject(X-Sub), where samples are split into training and testing groups with respect to subjects. (2) Cross-view(X-View) for NTU RGB+D 60 or Cross-Setup (X-Set) for NTU RGB+D 120, where action sequences are divided by their shooting cameras or camera setups. Kinetics Skeleton 400. The Kinetics Skeleton 400 Dataset [14] is sampled from the Kinetics-400 dataset by Openpose pose estimation toolkit [2]. It contains 240,436 training and 19,796 testing sequences. Every action sequence contains 18 joints wth their 2D locations and 1D confidence score at each time step.

Training Details
We use 8 STGABs in our experiments with their channels in {64, 64, 128, 128, 256, 256, 256 and 256} respectively. We randomly/uniformly sample 150 frames and then randomly/centrally crop 128 frames for training/test splits. The learning rate is 0.1 and divided by 10 at epochs {60, 90}, {60, 90}, and {80, 110} for NTU RGB+D Datasets, and Kinetics Skeleton 400 with total 120/180 epochs. The temporal convolution following STGAB owns kernel size 7. We combine our Separate Learning Strategy with multi-head mechanism and set one head as a single group. Our model is trained on a two-2080ti machine with batch size 32 and weight decay 0.0005 in PyTorch framework. Note that, no data augmentation technique is adopted for fair comparison.

Ablation Study
Model Effectiveness of proposed strategies. We incrementally build up the model with its individual components to verify their effectiveness and list the results in tab. 1. Here, SL denotes Separate Learning strategy and EMM means Explicit Motion Modeling. T DT W represents the learnable parameter to structurally weight edges while ATW denotes Adaptive Temporal Weighting. Compared to our baseline, directly building spatial-temporal graphs with STGAT causes 0.2% performance decrease. This demonstrates local feature redundancy severely hurts modeling short-term motion. We then testify the effectiveness of our proposed strategies for reducing local redundancy. Equipped with SL, STGAT obtains 0.6% performance boost. Through SL, STGAT deals with small groups of joints and is easier to distinguish vital connections. Combined with EMM, STGAT achieves 0.8% accuracy boost, which demonstrates EMM makes effect by capturing subtle motion instead of raw spare features. Next, we test the effect of Dynamic Temporal Weighting strategy. STGAT gains 0.8% performance boost from T DT W which structurally weights frames along temporal dimension. ATW also performs well and brings 0.7% performance boost. These results indicate our strategy corrects the equal weighting problem. We also try to combine T DT W with ATW to perform structural and adaptive weighting together, but don't witness further promotion. We speculate that they may overemphasize the weights. T DT W is adopted as default due to its high performance and low computation cost. In the following, we make a comparative test to verify the effectiveness of STGAT. We don't build spatial-temporal graphs but directly equip our baseline with proposed strategies. In this condition, SL and EMM will still make effect. As shown in tab. 1, it achieves 89.0% (+0.8%) but performs lower than STGAT (90.2%, +2.0%), which demonstrates STGAT can better capture local spatial-temporal dependencies for its local spatial-temporal modeling. Finally, we examine the effect of spatial-temporal modeling for STGAT. We compare it with spatial connections (baseline), dense connections among all joints in τ frames, and simply repeating spatial graph τ times along temporal dimension. Our STGAT outperforms all of them by a large margin (≥ 1.2%) which demonstrates the effectiveness of adaptive computation for edges in a local spatialtemporal neighborhood.
Study on model structure. We then test the choices of some hyperparameters in STGAT and give their results in tab. 1. We offer S, the number of heads, with four choices in {1, 4, 8 and 16}. We observe that larger S gives higher performance which indicates the effects of multiple built graphs, and the performance reaches top when S equals 8. For other STGAT configurations, like τ and d, we test different combinations of them. τ =1 means no temporal information is incorporated and only spatial modeling is performed. We first observe STGAT outperforms baseline when τ =1 and benefits from larger τ , which demonstrates the effect of local spatial-temporal dependencies. The gain of τ starts to decrease when it equals 5. We infer large τ makes similar effect with temporal-only modules and causes redundancy. Considering d, we observe accuracy drop when d increases, which shows closer frames are much more relevant in a local spatial-temporal neighborhood. We further construct two STGAT paths to make comparison with MS-G3D [25]. MS-G3D obtains performance boost by incorporating more paths for multiscale temporal information. But STGAT already earns better performance through one path with fewer computations and parameters than MS-G3D for its superior spatial-temporal modeling ability.
Effect on classifying similar actions. We demonstrate the effect of STGAT on classifying similar patterns in tab. 3. We compare STGAT with baseline and another spatial-temporal modeling method, MS-G3D, upon several similar action pairs from NTU RGB+D 60 Dataset. Compared with MS-G3D, STGAT outperforms it in all action pairs with up to 9.2% performance boost. MS-G3D fails to distinguish short-term motion with equal weights for neighboring joints, which introduces too much redundancy, even with two paths. Compared with baseline, STGAT obtains notable improvements, up to 12.5%, upon similar action pairs. These results verify the effectiveness of our STGAT on classifying similar action pairs owing to its adaptive spatial-temporal modeling. Please refer to the supplementary material for more examples and the full  Table 3. Class-wise Accuracy(%) Comparison for MS-G3D [25], baseline and STGAT on similar action pairs from NTU RGB+D 60 dataset with X-Sub Setting.
Comparison with previous methods with short-term modeling. We compare STGAT with MS-GCN [25] which assigns equal weights for local neighbors for spatial-temporal modeling. [25] also proposes MSTCN consisted of temporal convolutions with different dilations for multiscale temporal information. As shown in tab. 4, STGAT outperforms baseline as well as MSGCN by a large margin (+2.0%, +2.4%). When equipped with MSTCN, either baseline or MSGCN obtains satisfactory performance boost (+0.4%, +1.2%), accompanied by notable increase in parameters and GFLOPs. STGAT outperforms MSTCN hybrids by at least 1.2% with fewer parameters and GFLOPs and similar memory usage, which demonstrates the efficacy and efficiency of STGAT for capturing both short-term and long-term dependencies.  Table 4. Comparison with previous methods with short-term modeling on NTU RGB+D 60 dataset with X-Sub Setting.
Visualizations. We visualize the generated edges by STGAT, dense connections among all joints, and spatial connections (Baseline), in fig. 4 to give more analysis on the effect of STGAT. We select the action "hand waving" as an example. The left hand is the most important joint of action "hand waving" which forms the movement patterns of this action. Edges generated by STGAT   Table 5, 6 and 7 compare our final model with other state-of-the-art methods on three large-scale datasets NTU RGB+D 60 [27], NTU RGB+D 120 [27] and the Kinetics Skeleton 400 [14]. These methods include CNN-based methods [19], RNN-based methods [24,32], graph networks with predefined graph structure [20,25,28,41,7] and graph networks with adaptive adjacent matrix [30,29,31,45,42]. We first observe that most of graph-based methods gain better performance than non-graph methods by better modeling relationships. We then see graph methods with adaptive graph structure [30,29,31,45,42] or multi-scale feature aggregation [25] consistently perform better than previous methods, indicating the necessity to capture long-range dependencies. Our STGAT makes a step further to capture cross-spacetime information and outperforms all existing methods especially when MS-G3D aggregates spatial-temporal features with two paths.

Conclusion
In this paper, we propose a general framework, named STGAT, to model crossspacetime information. To reduce local spatial-temporal feature redundancy, we propose three simple modules on narrowing the scope of self-attention mechanism, dynamically weighting frames and separating subtle motion from static features. STGAT exceeds previous state-of-the-arts on three large-scale datasets and makes further progress over classifying similar action pairs, witnessed by both qualitative and quantitative results.

NTU RGB+D 120
Methods X-Sub X-Set SGN [45] 79.2 81.5 2s-AGCN [30] 82.9 84.9 Shift-GCN [7] 85.9 87.6 4s DSTA-Net [31]  where [·, ·] represents concatenation operation and W f is a weight vector projecting their concatenated result into a scalar. Besides, σ serves as an activation function that is usually ReLU or LeakyReLU in practice. Under the formulation defined above, a self-attention module can be implemented as shown in fig. 1 in practice. Here, θ(·) and ϕ(·) are instantiated as 1×1 convolutions and C e is adopted to reduce computation costs. g(·) = 1. Empirically, C e is defined as C out /r where r is an adjustable factor to reduce computations.

B Implementation for Adaptive Temporal Weighting Module
We give the implementation for the adaptive temporal weighting module in the fig. 2. We first perform 1-hop graph convolutions for current frame X t to aggregate local spatial features. Then, a 1 × 1 convolution kernel is utilized to enhance features of current frame X t . We subtract sampled features of regional neighbors from those of current frame to obtain motion features. Finally, after pooling along channel and temporal dimension, they are passed through a sigmoid function to weight frames in local spatial-temporal region.
C Class-wise Accuracy Comparison  Table 1: Class-wise Accuracy Comparison for MS-G3D [25], our baseline (DSTA-Net [31]) and STGAT on the Cross Subject setting of the NTU RGB+D 60 dataset, using the joint only. Values in the parentheses are accuracy differences compared to MS-G3D or baseline. Accuracy decrease is marked in red while accuracy increase is marked with green. STGAT outperforms MS-G3D and baseline in 38/60 and 46/60 action classes, respectively. Especially, on several similar action pairs like 'eat meal/snack' against 'brushing teeth', 'reading' against 'writing' and 'taking a selfie' against 'salute', STGAT makes great improvements.