As an emerging topic in face recognition, designing margin-based loss functions can increase the feature margin between different classes for enhanced discriminability. More recently, absorbing the idea of mining-based strategies is adopted to emphasize the misclassified samples and achieve promising results. However, during the entire training process, the prior methods either do not explicitly emphasize the sample based on its importance that renders the hard samples not fully exploited or explicitly emphasize the effects of semi-hard/hard samples even at the early training stage that may lead to convergence issues. In this work, we propose a novel Adaptive Curriculum Learning loss (CurricularFace) that embeds the idea of curriculum learning into the loss function to achieve a novel training strategy for deep face recognition, which mainly addresses easy samples in the early training stage and hard ones in the later stage. Specifically, our CurricularFace adaptively adjusts the relative importance of easy and hard samples during different training stages. In each stage, different samples are assigned with different importance according to their corresponding difficultness. Extensive experimental results on popular benchmarks demonstrate the superiority of our CurricularFace over the state-of-the-art competitors. Code will be available upon publication. The success of Convolutional Neural Networks (CNNs) on face recognition can be mainly credited to : enormous training data, network architectures, and loss functions. Recently, designing appropriate loss functions that enhance discriminative power is pivotal for training deep face CNNs. Current state-of-the-art face recognition methods mainly adopt softmax-based classification loss. Since the learned features with the original softmax is not discriminative enough for the open-set face recognition problem, several margin-based variants have been proposed to enhance features' discriminative power. For example, explicit margin, i.e., CosFace (Wang et al., 2018a) , Sphereface (Li et al., 2017) , ArcFace (Deng et al., 2019) , and implicit margin, i.e., Adacos (Zhang et al., 2019a) , supplement the original softmax function to enforce greater intra-class compactness and inter-class discrepancy, which are shown to result in more discriminate features. However, these margin-based loss functions do not explicitly emphasize each sample according to its importance. As demonstrated in Chen et al. (2019) , hard sample mining is also a critical step to further improve the final accuracy. Recently, Triplet loss (Schroff et al., 2015) and SV-Arc-Softmax (Wang et al., 2018b) integrate the motivations of both margin and mining into one framework for deep face recognition. Triplet loss adopts a semi-hard mining strategy to obtain semi-hard triplets and enlarge the margin between triplet samples. SV-Arc-Softmax (Wang et al., 2018b) clearly defines hard samples as misclassified samples and emphasizes them by increasing the weights of their negative cosine similarities with a preset constant. In a nutshell, mining-based loss functions explicitly emphasize the effects of semi-hard or hard samples. However, there are drawbacks in training strategies of both margin-and mining-based loss functions. For margin-based methods, mining strategy is ignored and thus the difficultness of each sample is not fully exploited, which may lead to convergence issues when using a large margin on small backbones, e.g., MobileFaceNet (Chen et al., 2018) . As shown in Fig. 1 , the modulation coefficient for the negative cosine similarities I(·) is fixed as a constant 1 in ArcFace for all samples during the entire training process. For mining-based methods, over-emphasizing hard samples in early training Figure 1: Different training strategies for modulating negative cosine similarities of hard samples (i.e., the mis-classified sample) in ArcFace, SV-Arc-Softmax and our CurricularFace. Left: The modulation coefficients I(t, cos θj) for negative cosine similarities of hard samples in different methods, where t is an adaptively estimated parameter and θj denotes the angle between the hard sample and the non-ground truth j-class center. Right: The corresponding hard samples' negative cosine similarities N (t, cos θj) = I(t, cos θj) cos θj + c after modulation, where c indicates a constant. On one hand, during early training stage (e.g., t is close to 0), hard sample's negative cosine similarities is usually reduced and thus leads to smaller hard sample loss than the original one. Therefore, easier samples are relatively emphasized; during later training stage (e.g., t is close to 1), the hard sample's negative cosine similarities are enhanced and thus leads to larger hard sample loss. On the other hand, in the same training stage, we modulate the hard samples' negative cosine similarities with cos θj. Specifically, the smaller the angle θj is, the larger the modulation coefficient should be. stage may hinder the model to converge. As SV-Arc-Softmax claimed, the manually defined constant t plays a key role in the model convergence property and a slight larger value (e.g., >1.4) may cause the model difficult to converge. Thus t needs to be carefully tuned. In this work, we propose a novel adaptive curriculum learning loss, termed CurricularFace, to achieve a novel training strategy for deep face recognition. Motivated by the nature of human learning that easy cases are learned first and then come the hard ones (Bengio et al., 2009) , our CurricularFace incorporates the idea of Curriculum Learning (CL) into face recognition in an adaptive manner, which differs from the traditional CL in two aspects. First, the curriculum construction is adaptive. In traditional CL, the samples are ordered by the corresponding difficultness, which are often defined by a prior and then fixed to establish the curriculum. In CurricularFace, the samples are randomly selected in each mini-batch, while the curriculum is established adaptively via mining the hard samples online, which shows the diversity in samples with different importance. Second, the importance of hard samples are adaptive. On one hand, the relative importance between easy and hard samples is dynamic and could be adjusted in different training stages. On the other hand, the importance of each hard sample in current mini-batch depends on its own difficultness. Specifically, the mis-classified samples in mini-batch are chosen as hard samples and weighted by adjusting the modulation coefficients I(t, cosθ j ) of cosine similarities between the sample and the non-ground truth class center vectors, i.e., negative cosine similarity N (t, cosθ j ). To achieve the goal of adaptive curricular learning in the entire training, we design a novel coefficient function I(·) that is determined by two factors: 1) the adaptively estimated parameter t that utilizes moving average of positive cosine similarities between samples and the corresponding ground-truth class center to unleash the burden of manually tuning; and 2) the angle θ j that defines the difficultness of hard samples to achieve adaptive assignment. To sum up, the contributions of this work are: • We propose an adaptive curriculum learning loss for face recognition, which automatically emphasizes easy samples first and hard samples later. To the best of our knowledge, it is the first work to introduce the idea of adaptive curriculum learning for face recognition. • We design a novel modulation coefficient function I(·) to achieve adaptive curriculum learning during training, which connects positive and negative cosine similarity simultaneously without the need of manually tuning any additional hyper-parameter. • We conduct extensive experiments on popular facial benchmarks, which demonstrate the superiority of our CurricularFace over the state-of-the-art competitors. Comparison with ArcFace and SV-Arc-Softmax We first discuss the difference between our CurricularFace and the two competitors, ArcFace and SV-Arc-Softmax, from the perspective of the decision boundary in Tab. 1. ArcFace introduces a margin function T (cos θ yi ) = cos(θ yi + m) from the perspective of positive cosine similarity. As shown in Fig. 4 , its decision condition changes from cos θ yi = cos θ j (i.e., blue line) to cos(θ yi + m) = cos θ j (i.e., red line) for each sample. SV-Arc-Softmax introduces additional margin from the perspective of negative cosine similarity for hard samples, and the decision boundary becomes cos(θ yi + m) = t cos θ j + t − 1 (i.e., green line). Conversely, we adaptively adjust the weights of hard samples in different training stages. The decision condition becomes cos(θ yi +m) = (t+cos θ j ) cos θ j (i.e., purple line). During the training stage, the decision boundary for hard samples changes from one purple line (early stage) to another (later stage), which emphasizes easy samples first and hard samples later. Comparison with Focal loss Focal loss is a soft mining-based loss, which is formulated as: β , where α and β are modulating factors that need to be tuned manually. The definition of hard samples in Focal loss is ambiguous, since it always focuses on relatively hard samples by reducing the weight of easier samples during the entire training process. In contrast, the definition of hard samples in our CurricularFace is more clear, i.e., mis-classified samples. Meanwhile, the weights of hard samples are adaptively determined in different training stages. In this paper, we propose a novel Adaptive Curriculum Learning Loss that embeds the idea of adaptive curriculum learning into deep face recognition. Our key idea is to address easy samples in the early training stage and hard ones in the later stage. Our method is easy to implement and robust to converge. Extensive experiments on popular facial benchmarks demonstrate the effectiveness of our method compared to the state-of-the-art competitors. Following the main idea of this work, future research can be expanded in various aspects, including designing a better function N (·) for negative cosine similarity that shares similar adaptive characteristic during training, and investigating the effects of noise samples that could be optimized as hard samples.
A novel  Adaptive Curriculum Learning loss for deep face recognition
{"src": "scitldr-AIC_train", "paper_id": "B1eksh4KvH"}