Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble

Sign language is commonly used by deaf or mute people to communicate but requires extensive effort to master. It is usually performed with the fast yet delicate movement of hand gestures, body posture, and even facial expressions. Current Sign Language Recognition (SLR) methods usually extract features via deep neural networks and suffer overfitting due to limited and noisy data. Recently, skeleton-based action recognition has attracted increasing attention due to its subject-invariant and background-invariant nature, whereas skeleton-based SLR is still under exploration due to the lack of hand annotations. Some researchers have tried to use off-line hand pose trackers to obtain hand keypoints and aid in recognizing sign language via recurrent neural networks. Nevertheless, none of them outperforms RGB-based approaches yet. To this end, we propose a novel Skeleton Aware Multi-modal Framework with a Global Ensemble Model (GEM) for isolated SLR (SAM-SLR-v2) to learn and fuse multi-modal feature representations towards a higher recognition rate. Specifically, we propose a Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics of skeleton keypoints and a Separable Spatial-Temporal Convolution Network (SSTCN) to exploit skeleton features. The skeleton-based predictions are fused with other RGB and depth based modalities by the proposed late-fusion GEM to provide global information and make a faithful SLR prediction. Experiments on three isolated SLR datasets demonstrate that our proposed SAM-SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins. Our code will be available at https://github.com/jackyjsy/SAM-SLR-v2


I. INTRODUCTION
Sign language [1] is a visual language performed with the dynamic movement of hand gestures, body posture, and facial expressions. It is a widely-used alternative approach for deaf and mute people to communicate effectively. Understanding and performing sign language requires a substantial time of learning which is beyond feasible for the public, which leads to a barrier between deaf-mute people and others. Moreover, sign language is dependent on language [2], [3], [4] (e.g., English and Chinese) and culture [5] (e.g., American and British) that further limits its popularity. Sign Language Recognition (SLR) aims to help deaf-mute people communicate smoothly with others in their daily life by automatically interpreting sign language. As machine learning and computer vision make great progress in the past decade, SLR has drawn much research attention.
SLR contains two tasks, isolated SLR and continuous SLR. The isolated setting is a fine-grained and fully supervised classification at word (gloss) level, while the continuous setting maps whole videos into sentences (i.e., sequences of glosses) in a weakly supervised manner. SLR is a more challenging problem compared with conventional action recognition for the following reasons. First, sign language requires both delicate hand gestures and global body motion to distinctly and accurately express its meaning. Facial expressions may also be used to express emotions and emphasis. Similar gestures may sometimes express different and even opposite meanings. Second, different signers may perform sign language differently (e.g., left-hander, right-hander, different speed, body shape, and localism), which makes SLR more challenging. Collecting more samples from as many signers as possible is desired yet expensive. To this end, it requires a finer-grained model to capture the delicate dynamics of the whole human body. Besides, we expect the model to effectively merge the information from different cues when making the final recognition.
Recently, in human action recognition, skeleton-based approaches have become more popular and drawn increasing attention due to their strong adaptability to dynamic circumstances and complicated backgrounds [19], [20], [21], [22], [23]. Besides, the skeleton-based methods provide complementary information to RGB-based modalities, hence their ensemble results further improve the overall recognition rates. However, there exist some barriers that hinder the extension of skeleton-based methods to the SLR task. The skeleton-based methods for action recognition rely on ground-truth annotations of human bodies provided by motion capture systems, which restrict themselves to limited datasets captured in labcontrolled environments. Apart from that, most motion capture systems only focus on the fewer body keypoints excluding hand keypoints. The provided keypoints are insufficient to recognize sign language performed with delicate hand gestures and motions. Some researchers attempt to obtain hand poses using separate hand detectors with keypoint estimators and propose to use an RNN-based model to recognize the sign language [24]. Unfortunately, their estimated hand keypoints are unreliable, and the RNN-based model cannot learn the dynamics of the human skeleton properly.
In this work, we focus on the isolated SLR task. We propose a Skeleton Aware Multi-modal SLR framework with a Global Ensemble Model (SAM-SLR-v2) to explore the potential of skeleton-based SLR and fuse with other modalities in both RGB and RGB-D scenarios to further improve the recognition rate. Specifically, we design a new spatio-temporal skeleton graph for SLR using whole-body keypoints extracted by a pretrained whole-body pose estimator. Then we propose a multi-stream Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics. To fully exploit the information in whole-body keypoints, we propose a novel Separable Spatial-Temporal Convolution Network (SSTCN) to learn from the whole-body skeleton features. Moreover, studies on action recognition reveal that data from different modalities complement each other, provide knowledge of latent correlation, and further improve the final performance. Although we can simply add predictions from all modalities together to achieve higher accuracy, we desire a method that tunes the best weight for each modality in a data-driven way. Hence, we propose a Global Ensemble Model (GEM) to automatically learn the multi-modal ensemble and improve the overall recognition rate. Our main contributions can be summarized as follows: • We construct novel 2D and 3D skeleton graphs designed for SLR using a pretrained whole-body pose estimator and graph reduction, which requires no extra annotation effort. • We propose a novel SL-GCN to model dynamics in the skeleton graphs. To our best knowledge, this is the first successful attempt to tackle the SLR task using 2D/3D whole-body skeleton graphs that surpasses RGB-based methods. • We propose a novel SSTCN to further exploit wholebody skeleton features. It can significantly improve the accuracy compared with the traditional 3D convolution.
• We propose an ensemble model GEM for both RGB and RGB-D based SLR, which learns weights from seven modalities and achieves state-of-the-art performance on three isolated SLR datasets with significant margins. A preliminary version of this work [25] has been reported in the corresponding workshop of the SLR challenge [26], during which we won the championships in both RGB 1 and RGB-D tracks 2 . Compared with our workshop version, we have made the following improvements: (1) We introduce a new modality Keypoint3D, which considers 3D coordinates in space and deal with occlusions. It improves the overall recognition rate of RGB-D ensemble. (2) We propose a new learning-based late-fusion ensemble method named GEM for multi-modal ensemble, which achieves higher recognition rates than our previous version and saves efforts on weights tuning.
(3) The test labels of the AUTSL dataset has been released, so we update our performance from the validation set to the test set. (4) Beyond the challenge dataset (AUTSL), we report our performance on two additional large-scale datasets for isolated (i.e., SLR500 and WLASL2000) compared with the recent state-of-the-art methods. Our approach significantly surpasses their performance with notable margins. (5) We update our figures, provide more details of models, analyze ensemble sensitivity, and discuss challenging cases, which may inspire future research on SLR.
II. RELATED WORK Sign Language Recognition (SLR) achieves significant progress in obtaining high recognition accuracy in recent years due to the development of practical deep learning architectures and the surge of computational power [15], [10], [13], [27]. Researchers have been modeling the spatio-temporal information in the videos using 2D CNN with RNN or 3D CNN. A 3D-convolutional SLR network associated with attention modules is proposed in [15] to learn the spatio-temporal features from raw videos. CNN, Feature Pooling Module, and LSTM Networks associated with adaptive weights are utilized in [10] to obtain distinctive representations. Besides, researchers have been extending isolated SLR to weakly supervised continuous SLR and Sign Language Translation (SLT) by incorporating sequence learning methods [28], [29], [14], [30], [31]. Recently, multi-cue methods using upperbody poses, hand keypoints, and mouth features have been developed to improve the recognition rate. For example, hand pose priors are introduced in hand-aware frameworks for further performance improvement in [17], [16]. A spatialtemporal multi-cue network for continuous SLR and SLT is proposed in [18]. However, these methods are still not effective enough to capture the complete motion information for robust sign language recognition. Skeleton-based Action Recognition mainly focuses on learning dynamic patterns from the motion of the human skeleton and effectively recognizing human action and activities [32], [33], [34], [35], [36], [37]. In the meanwhile, aiming for a higher recognition accuracy, it can collaborate with other Fig. 2. Construction of 2D and 3D graphs. A pretrained whole-body pose estimator is applied on an input RGB video (a) to obtain 133-point whole-body keypoints (b). We then construct a spatio-temporal 2D graph following the natural connections of the human body. The graph is reduced to 27-node (c) using graph reduction to mitigate noises. Each node in the 2D graph is represented by (x, y, s) where x-y are 2D coordinates and s is the confidence score. The 3D graph is obtained by overlaying the 2D coordinates on the depth video, reading the depth encoding z at the corresponding location, and treating it as an additional dimension as (x, y, z, s).
modalities (e.g., RGB, depth, and EMG) and benefit from multi-modal learning [38], [39], [40], [41], [42]. Recurrent neural networks (e.g., RNN and LSTM) are once popular in modeling the temporal information of skeleton data [33], [37], [36], [43]. Recently, a graph-based approach that implements a Graph Convolutional Network (GCN) to model the dynamic patterns in skeleton data is first explored by ST-GCN [19]. It draws much research attention that a few improved models are developed to further improve the performance [35], [44], [45], [46], [21], [22], [23]. Specifically, AS-GCN [35] digs the latent joint connections to boost the recognition performance. A twostream approach is presented in [45] and further extended to four streams in [21]. DecoupleGCN [22] increases the GCN capacity while introducing no extra computational cost. Inspired by ResNet [47], ResGCN [23] incorporates a bottleneck architecture to boost model capacity, reduce parameters, and avoid gradient vanishing. However, skeleton-based SLR methods are still under exploration. Simply applying the ST-GCN to SLR has been unsuccessful, which only reaches around 60% top-1 accuracy on 20 classes (much worse than RGB-based approaches) [48]. Multi-modal Approach aims to explore data captured from different resources, by different devices, and from distinctive views to improve the overall performance. Its motivation lies in an assumption that different modalities contain unique and view-specific information that complements each other.
The assumption suggests that multi-modal information can be fused together to further boost performance. Frameworks that learn robust and view-invariant feature representation for downstream tasks are proposed in [49], [50]. A weight-sharing model is developed in [51] to obtain modality hallucination for multi-modal image classification. DA-Net [52] adopts a view-independent module collaborated with a view-specific module to capture the multi-modal information and effectively merges the prediction scores. PM-GANs [53] utilizes a crossview generative strategy associated with a novel fusion to learn the prediction correlations effectively. A feature factorization network is proposed in [54] which learns the specific viewshared information for RGB-D action recognition. A cascaded residual autoencoder is designed to handle incomplete view classification settings [55]. Researchers also propose to use a super vector to fuse the multi-modal representations [56]. En-couraged by the success of those multi-modal approaches, we aim to incorporate more modalities (e.g., visual, depth, body gesture, and hand gesture) to capture both view-dependent and view-specific information from all aspects. We aim to learn via a universal framework to achieve higher performance.

III. METHODOLOGY
This section first gives an overview of the proposed SAM-SLR-v2 framework. Then we introduce the SL-GCN for keypoint graphs and the SSTCN for skeleton features, respectively. After that, we present an effective 3DCNN baseline model for the other modalities. Last, we describe the latefusion GEM, for the multi-modal ensemble.

A. SAM-SLR-v2 Framework Overview
An overview of the proposed SAM-SLR-v2 is shown in Figure 1. Seven modalities processed from RGB and depth videos are considered in the proposed framework. Three different architectures (i.e., SL-GCN, SSTCN, and 3DCNN) are used to extract features from the seven modalities and make predictions of sign language glosses independently. The latefusion ensemble model (i.e., GEM) takes predictions from all the modalities and outputs final predictions in the RGB and the RGB-D scenarios.

B. SL-GCN for Skeleton Keypoints
Graph Construction and Reduction. For skeleton-based action recognition, researchers rely on ground-truth skeleton keypoints annotated by motion capture systems such as Kinect v2 [57], which unfortunately is not capable of providing hand and finger annotations. Since hand gestures play a crucial role in performing sign language, we use a pose estimator pretrained with whole-body annotations to predict whole-body keypoints, which include 133 landmarks of face, body, hands, and feet. A spatial 2D graph can then be constructed by connecting every pair of adjacent keypoints according to the human-body natural connections. This graph is further extended to a spatio-temporal graph by connecting all the nodes to themselves in the temporal dimension. Mathematically, the , the minimum number of nodes) between Node v i and v j in the shortest path. However, contrasting with the graph used in skeleton-based action recognition which contains around 17 nodes, the wholebody skeleton graph contains too many nodes and edges which introduce high-level unexpected noise. Besides, if the distance between two nodes is too far (i.e., have many nodes in between), it is inaccurate to explore their interactions. Our experiment shows that simply using the whole-body skeleton graph results in lower accuracy. Therefore, we operate a graph reduction on the whole-body skeleton graph which trims down the 133 nodes to 27 nodes based on our observations of the videos and visualizations of the GCN activation. The resulted graph consists of seven nodes for the upper body (nose, eyes, shoulders, and elbows) and ten nodes for each hand, as illustrated in Figure 2(c). Graph reduction leads to faster model convergence and significantly higher recognition rates. Each node in the 2D graph is represented by (x, y, s) where x-y are 2D coordinates and s is the confidence score. When depth information is available, we construct a 3D graph by reading the corresponding depth z at keypoint locations x-y and treating it as an additional dimension as (x, y, z, s), as illustrated in Figure 2(d). Graph Convolution. We adopt the spatio-temporal graph convolution with the spatial partitioning strategy [19] to capture the embedded dynamics in the whole-body skeleton graph. We implement the spatial graph convolution as where A represents an adjacent matrix of intra-body connections, I represents an identity matrix of self-connections, D stands for the diagonal degree of (A + I), and W is the trainable weights of the convolution. Practically, Equation 2 is implemented by performing standard 2D convolution and multiplying the results by D − 1 2 (A + I)D − 1 2 . To perform temporal graph convolution, we implement a standard 2D convolution on the temporal dimension with a kernel size k t ×1 as the reception field. We adopt an extended variation of the spatial GCN called decoupling GCN [22]. In a decoupling GCN layer, to further boost the model capacity, the extracted features are grouped into G groups that each group has its trainable adjacent matrix A. We then concatenate the outputs of all groups back together as the output features. Multi-stream SL-GCN. Inspired by [21] which adopts a multi-stream workflow for action recognition, we find that it is also worth investigating 1st-order coordinates (joints), 2ndorder vector (bone vector), and their motion vectors for sign language recognition, as shown in Figure 3(a). Following the natural connections of the human body, we generate the bone vectors pointing from the starting joints to their ending joints. Thus the bone stream is represented by a tree graph where the nose acts as the root node. Mathematically, the starting and ending joints can be represented as where (x, y, [z], s) represents 2D coordinates, optional depth, and the confidence score. The bone vectors v B are calculated by subtracting Equation 3 from Equation 4 as where the set H contains all natural connections of the human body. Motion streams are obtained by subtracting the difference between adjacent frames. The joint motion vectors v JM and the bone motion vectors v JM are represented as We train every stream separately, multiply their predictions by assigned weights, and summing up the results as the final prediction. SL-GCN Structure. The structure of our proposed SL-GCN is presented in Figure 3(b). The input stream is permuted and normalized before feeding into N instances of SL-GCN units for spatio-temporal graph convolution. An average pooling is then applied to the temporal dimension of the resulted features. The result is reshaped and fed into a fully connected layer for classification. Our proposed SL-GCN unit is illustrated in Figure 3(c). We find that deep graph models are easier to overfit on the video classification task and the ordinary dropout layer works poorly in GCNs. We propose to construct the basic SL-GCN Unit with a decoupled spatial convolutional layer (DecoupleSCN) [22] to mitigate overfitting. We also introduce an STC (spatial, temporal, and channel-wise) self-attention mechanism inspired by [21]. As illustrated in Figure 3(d), the STC module consists of modules of spatial attention, temporal attention, and channel attention connected in a cascaded configuration. In our experiment, we use N = 10 such SL-GCN units in the proposed SL-GCN.

C. SSTCN for Skeleton Features
We propose an SSTCN model to exploit the whole-body features in addition to the keypoint coordinates. We can learn from ResNet2+1D [58] that the performance of an action recognition model can be further improved via factorizing the network into a temporal part and a spatial part. Therefore, our SSTCN model is separated into four stages to handle the features from different dimensions. The pipeline is shown in Figure 4. We save the features of 33 keypoints including 1 nose keypoint, 4 mouth keypoints, 6 upper-body keypoints, and 22 hands keypoints before the argmax operation in the pose estimator. We then uniformly sample 60 frames in each video for SSTCN for recognition. The saved features are resized to 24 × 24 using maximum pooling. We process the input features with a 2D separable convolution layer that reduces the parameters and converges easily. At Stage one, features are reshaped from 60×33×24×24 to 60×792×24, and fed to 1×1 convolution layers for temporal convolution. At Stage two, the extracted features are then shuffled and grouped into 60 groups for 3×3 grouped convolution to extract spatial features of each frame. At the next stage, we start processing the features in the feature dimension. Specifically, the features are shuffled again and grouped into 33 groups. We use grouped 3×3 convolution on spatial and temporal dimensions for each keypoint. At the last stage, a few fully connected layers are used to make final predictions. We adopt a residual path for the first three stages to avoid gradient vanishing. Besides, random dropouts are deployed in every module to avoid overfitting [59].
Researchers have found that one-hot labels and crossentropy loss may be easy to overfit with limited data [60]. So we use the technique of label smoothing to avoid overfitting. Mathematically, label smoothing is defined as where k is the number of classes, q (k|x) is a label-smoothed predicted distribution, is a hyper-parameter in (0, 1), and u(·) represents the uniform distribution. The cross-entropy loss is then modified as where H(·) is the cross-entropy function and q is the real data distribution, and p stands for the predicted distribution. The modified cross-entropy can be explained as the penalty to the difference between the predicted distribution and the real distribution combined with the difference between the predicted distribution and a prior distribution (e.g., uniform distribution). To further improve the performance, we replace all activations to the Swish [61] activation function as

D. 3DCNN Baselines for the Other Modalities
As mentioned in Section II, studies reveal that ensembles from multiple cues and modalities could further improve the overall performance. To benefit from the other modalities (i.e., RGB frames, optical flows, HHA, and depth flow), we hence build a simple yet effective baseline using 3D CNNs. Most 3D CNN architectures are easy to overfit, especially on smaller datasets. In our experiment, The ResNet2+1D [58] architecture that separates the temporal and spatial convolution of 3D CNNs, provides the best result compared with other popular 3D CNN networks. We also notice that deeper model depth does not always lead to better performance while making the network to overfit easier. So we choose the ResNet2+1D-18 variation pretrained on the Kinectics dataset [62] as our backbone. Apart from that, to improve the accuracy further, we pretrain the model on the largest available SLR dataset SLR500 [63] for RGB frames. Pretraining increases the final accuracy by around 1% and improves the model convergence. Similar to the SSTCN presented in Section III-C, we adopt the Swish activation described in Equation 10 instead of using ReLU. Besides, to mitigate overfitting, we implement the label smoothing technique in Equation 8 with the corresponding cross-entropy loss in Equation 9.

E. Multi-modal Late-fusion Ensemble
Model-free Late Fusion. In our previous version, we use a simple late-fusion approach to fuse predictions from all modalities. To be specific, for all modalities, we save the predictions from the last fully connected layers before the softmax. We assign weights to all modalities manually and add them up as the final prediction q RGB-D =α 1 q skel3D + α 2 q RGB + α 3 q flow + α 4 q feat where q represents the predictions before softmax, α 1,2,3,4,5,6 are hyper-parameters to be tuned based on validation accuracy.  as illustrated in Figure 6. The whole process can be described as where q represents the results of each modality, G represents the prediction procedure of GEM, and [·] stands for the concatenation operation. The first layer of the model is used to down-sample the modalities effectively with a global convolution. Therefore, the kernel of the first layer is C × 1.
Then we use 1 × 1 fully connected layers to predict the final weights of all modalities. The final weights are multiplied with the inputs respectively and the weighted modalities are summed up as the final prediction.

IV. EXPERIMENTS
In this section, we report the evaluation results of our proposed SAM-SLR-v2 on three major sign language recognition datasets of different sign languages. We show the singlemodality performance of our proposed multi-stream SL-GCN, SSTCN, and 3DCNN models, as well as different combinations of their ensembles. We demonstrate the effectiveness of proposed approaches via ablation studies on the components of SL-GCN, SSTCN and 3DCNN networks. We also study the proposed GEM compared with simple model-free late fusion. Last, we show some challenging cases and discuss the limitations of SAM-SLR-v2.
A. Datasets AUTSL Dataset [64] is a Turkish SLR dataset collected using Kinect V2 sensor [57], [67]. Statistically, 226 different sign glosses are performed by 43 signers with 20 backgrounds. The dataset contains 38,336 videos that split into training, validation, and testing subsets. We use their currently released balanced test set in our experiments and report both top-1 and top-5 recognition rates. SLR500 Dataset [63] is a balanced Chinese sign language dataset for isolated sign language recognition (sometimes referred to as CSL isol.) that contains 500 words performed by 50 signers. Each word is performed by all 50 signers 5 times, so there are 125,000 videos in total. This dataset is captured in a controlled lab environment with a solid-color background. The former 36 signers are used for training and the later 14 signers are used for testing. WLASL2000 Dataset [11] is a American Sign Language with a vocabulary size of 2000 words. It is a challenging dataset collected from web videos performed by 119 signers. It contains 21,083 samples with unconstrained recording conditions. The dataset is imbalanced and the average samples per video are much lower than the above two datasets. We follow the signer-independent settings for all the above datasets. Besides, we are also aware of other large-scale isolated sign language datasets. Microsoft American Sign Language dataset (MS-ASL) [65] is an American sign language dataset that is no longer publicly available due to the deletion of the YouTube-hosted videos. A new large-scale British Sign Language dataset (BSL-1K) [66] has not been released to the public yet at the moment of writing. A statistical summary of the mentioned datasets is provided in Table I.

B. Multi-modal Data Preparation
Whole-body Pose Keypoints and Features. MMPose [68] provides a whole-body pose estimator pretrained with wholebody keypoint annotations and HRNet [69] as its backbone. We use the pretrained model to obtain 133 keypoints from RGB videos, based on which we construct the 27-node 2D graph in Section III-B and process the 3D graphs using depth videos. Keypoint coordinates are normalized to [-1,1]. Random sampling, mirroring, rotating, scaling, jittering, and shifting are applied as data augmentations. We use a sample duration of 150 frames. We repeat videos with lesser than 150 frames to 150 frames. To obtain skeleton features for each video, we downsample the estimated heatmaps, choose 33 joint channels, and uniformly sample 60 frames. RGB Frames and Optical Flow. We extract all frames from RGB videos to load and process them faster in parallel. Besides, we use the Denseflow API implemented with OpenCV and CUDA that provided by OpenMMLab [70] to obtain the optical flows using the TVL1 algorithm [71]. The output x and y flow maps are concatenated in the channel dimension. We cropped all RGB and optical flow frames to the bounding boxes from the pose estimator, and then resize them to 256 × 256. When we train the model, we sample 32 consecutive frames randomly from the input video. When testing, we sample five such clips uniformly from the input videos and average on their predicted scores. Depth HHA and Depth Flow. HHA stands for the horizontal disparity, height above the ground, and angle normal that encode a depth map into a three-channel RGB image. Because HHA features enable better scene understanding, we extract HHA features from depth videos as another modality instead of inputting gray-scale depth maps directly. The depth videos of the AUTSL dataset come with masks, so we mask out those regions and fill them with zeros when generating HHA. An HHA example is shown in Figure 5(c), where black regions are masked out pixels. Cropping and resizing operations are performed the same way as RGB frames. We also apply the same data augmentations to HHA features as the RGB modality. Apart from that, we use Denseflow to extract optical flow from the depth modality as well. A sample of depth flow is shown in Figure 5(e). Compared with optical flows (e.g., Figure 5(d)), depth flows contain less noise and capture distinctive motion information.

C. Performance of Multi-stream SL-GCN
Both top-1 and top-5 recognition rates of the proposed multi-stream SL-GCN are reported in Table II. Among the four streams, the joint stream provides the best accuracy. The multi-stream approach further improves the overall accuracy, which demonstrates that our proposed whole-body skeleton graph and multi-stream SL-GCN are very effective. Table IV shows that SL-GCN (Keypoints 2D and 3D) perform the best in all single-modality methods. Since the skeleton graphs are less complicated, the graph-based method is lighter-weight and much faster to inference compared with large-capacity 3DCNN-based models, which is another advantage of the proposed SL-GCN. Table III presents ablation studies on the proposed SL-GCN. We find that the graph reduction technique contributes the most to the recognition rate. Without that, it can barely learn from the noisy dynamics in the 133-node skeleton graph. Due to the limitation of annotated data, the model is easy to overfit, thus the data augmentation techniques are also important in learning the embedded dynamics. Besides, we find that the DropGraph module, the decoupling GCN module, and the STC attention mechanism all contribute to the final performance.

D. Multi-modal Performance on AUTSL Dataset
The results of all single-modal methods on the AUTSL balanced test set are reported in Table IV. The Keypoints 2D and Keypoints 3D method represents our proposed multistream SL-GCN using 2D and 3D skeleton graphs. They perform the best compared with the other methods. We also find that the depth flow achieves a little better recognition rate compared with the RGB flow resulted from lesser noisy data. The fused ensembles in both RGB and RGB-D scenarios using different choices of modalities are summarized in Table V as three groups. The skeleton-based ensemble (Skel 2D and Skel 3D) achieves better accuracy than "RGB + Flow" and Depth ensemble (HHA + Depth Flow), which demonstrates that the proposed skeleton-based methods are very effective in SLR. The RGB and RGB-D ensemble results that use all available modalities show that the skeleton-based methods also complement RGB and depth modalities. Their collaboration further improves the overall performance. Comparisons with other players in the challenge are reported in Table VI. Our improved SAM-SLR-v2 achieves better recognition rates compared with its previous version SAM-SLR that won the championships of the SLR challenge. During the challenge, we are allowed to use the validation labels to fine-tune our models.
Our results with fine-tuning on the validation set are shown at the bottom. Our results without fine-tuning still surpass the other methods fine-tuned with validation labels. Our method achieves the state-of-the-art with significant margins in both RGB and RGB-D scenarios.

E. Multi-modal Performance on SLR500 Dataset
Performance on the isolated Chinese sign language dataset (SLR500) is reported in Table VII. Since this dataset is collected in a lab-controlled environment with a large number of repetitions and invariant background, all modalities obtain higher recognition rates compared with the other two datasets. For the same reason, compared with skeleton-based methods, RGB-based 3DCNN achieves a higher Top-1 recognition rate but a lower Top-5 recognition rate. Since the SLR500 dataset

Modality
Top-1 Top-5 GLE-Net [73] 96.80 -Hand + RGB [16] 98. provides only a validation set instead of train-val-test splits, we use the weights learned from the AUTSL dataset to obtain the final ensemble results. Compared with the recent state-ofthe-art method GLE-Net [73] and Hand+RGB [16], our twomodality methods and all-modality SAM-SLR-v2 all achieve much better recognition rates.

F. Multi-modal Performance on WLASL2000
The WLASL2000 dataset is the most challenging variation of the WLASL dataset with unconstrained backgrounds and camera conditions. Since the dataset is not balanced, we report both per-instance accuracy as well as per-class accuracy following [65], [16]. As reported in Table VIII, the keypoints modality performs the best (51.50% per-instance) among all the single-modal methods due to its independence on backgrounds and resistance to noises. The keypoints and RGB modalities complement each other and boost the twomodal performance. Using all the available modalities, the top-1 performance is improved to 57.55% per-instance. Our proposed late ensemble model is capable of raising the recognition rates by around 0.7%. Compared with other state-ofthe-art methods, our proposed SAM-SLR-v2 achieves the best recognition rates in both per-instance and per-class metrics with large leading gaps of around 8%.

G. Ablation Study on SSTCN and 3DCNN
An ablation study on SSTCN is provided in Table IX. Compared with ResNet3D [74], [75] and ResNet2+1D [58] on  the same feature size, SSTCN is more effective on recognizing sign glosses using skeleton features. Our study also reveals that the proposed SSTCN can achieve even higher accuracy with larger input feature sizes. Table X shows an ablation study of 3D CNN architecture using the RGB frames. It shows that both swish activation and label smoothing are effective, where they improve the top-1 accuracy by 2% and 1%, respectively. Pretraining on the CSL dataset [63] improves the overall accuracy by 1.4%. The ResNet2+1D-18 backbone provides 1.7% better performance than ResNet3D-18.

H. Comparison between Multi-modal Ensembles
The model-free simple ensemble method requires a lot of effort in tuning the weights of different modalities. With our proposed late-fusion GEM, those weights are automatically learned from the data. We study the effectiveness of GEM compared with the simple ensemble method. In Table VI, VII, and VIII, performance of model-free simple ensemble is shown as SAM-SLR. Compared with the simple ensemble, our proposed late-fusion GEM can improve the overall performance as well as avoid hard tuning on modality weights.

I. Sensitivity Analysis of Ensemble Methods
A sensitivity study on ensemble methods is conducted on the AUTSL test dataset. In the experiment, we use the manually tuned parameters in Section III-E as our basis and change one parameter at a time (varying from 0.0 to 2.0), while keeping the other parameters fixed. We plot the resulted top-1 accuracy to analyze the sensitivity of those ensemble parameters. In the RGB scenario, we have four modalities and we analyze the sensitivity of α 1 to α 4 , as shown in Figure 7(a). In the RGB-D scenario, we analyze the sensitivity of α 1 to α 6 , as shown in Figure 7(b). In this experiment, the ensemble parameters tuned on the validation set do not provide the highest performance on the test set. The ensemble performance is sensitive to the values of the parameters that small changes may lead to large variations in the ensemble accuracy. Moreover, when there are more modalities (e.g., in the RGB-D scenario), it is hard to optimize all those parameters manually. Our proposed late-fusion GEM solves this problem by automatically learning these parameters from data. Figure 7(c) shows the top-1 accuracy during training the proposed model. Initially, the ensemble model gives sensitive results due to its large search space. The model converges after 100 epochs and delivers steady and accurate ensemble results. In summary, the proposed late-fusion GEM is very effective in learning the multi-modal ensemble, gives robust higher performance than simple ensemble, and saves a lot of effort to tune the best parameters. Figure 8 shows some challenging cases of sign language recognition from the AUTSL dataset. The offline full-body pose estimator may fail due to off-screen or occlusion, especially for fingers, which is a common issue for pose estimation methods. However, fingers play a critical role in expressing signs. Some of those failures can be corrected by RGBbased features. That is the reason why multi-modal ensemble significantly boosts the recognition rate. Moreover, a same sign may be performed very differently by different signers (e.g., signers may use their left hands, right hands, or both hands to perform the same gloss). Hence, mirror augmentation is very important in model training, but more data collected from more signers are desired. Last, opposite signs could be visually similar in a single frame (e.g., heavy vs light-weight). Distinguishing between those signs requires delicate modeling of spatio-temporal dynamics. In our experiment, we find that skeleton-based methods are smarter choices over RGB-based methods.

V. CONCLUSION
In conclusion, we propose a novel SAM-SLR-v2 framework to learn multi-modal feature representations from RGB-D videos towards more effective and robust isolated SLR. Among those modalities, our proposed skeleton-based methods are the most effective in modeling motion dynamics due to their signer-independent and background-independent characteristics. Specifically, we construct novel 2D and 3D spatio-temporal skeleton graphs using pretrained whole-body keypoint estimators and propose a multi-stream SL-GCN to model the embedded motion dynamics. Our method does not require any additional effort on annotating hands and provides more reliable hand keypoint estimation than off-line hand detectors. Besides modeling motion dynamics of keypoint coordinates via graphs, we propose SSTCN to predict using skeleton features. Furthermore, we study the multi-modal fusion problem based on the other modalities (i.e., RGB frames, optical flow, HHA, and depth flow) via a learning-based late-fusion ensemble model named GEM. Experimentally, we show that our proposed SAM-SLR-v2 framework achieves the state-of-the-art performance on three challenging datasets for isolated SLR (i.e., AUTSL, SLR500, and WLASL2000) as well as won the championships in both RGB and RGB-D tracks during the CVPR 2021 challenge on isolated SLR. We hope our work could facilitate and inspire future research on SLR.