Self-Paced Deep Regression Forests with Consideration of Ranking Fairness

Deep discriminative models (DDMs), e.g. deep regression forests and deep decision forests, have been extensively studied recently to solve problems such as facial age estimation, head pose estimation, etc.. Due to a shortage of well-labeled data that does not have noise and imbalanced distribution problems, learning DDMs is always challenging. Existing methods usually tackle these challenges through learning more discriminative features or re-weighting samples. We argue that learning DDMs gradually, from easy to hard, is more reasonable, for two reasons. First, this is more consistent with the cognitive process of human beings. Second, noisy as well as underrepresented examples can be distinguished by virtue of previously learned knowledge. Thus, we resort to a gradual learning strategy -- self-paced learning (SPL). Then, a natural question arises: can SPL lead DDMs to achieve more robust and less biased solutions? To answer this question, this paper proposes a new SPL method: easy and underrepresented examples first, for learning DDMs. This tackles the fundamental ranking and selection problem in SPL from a new perspective: fairness. Our idea is fundamental and can be easily combined with a variety of DDMs. Extensive experimental results on three computer vision tasks, i.e., facial age estimation, head pose estimation, and gaze estimation, show our new method gains considerable performance improvement in both accuracy and fairness. Source code is available at https://github.com/learninginvision/SPU.


I. INTRODUCTION
D EEP discriminative models (DDMs) have been extensively studied recently to solve a variety of computer vision problems, with applications in image classification, action recognition, and object detection, to name a few. DDMs use deep neural networks (DNNs) [1]- [4] to formulate the input-output mapping in discriminative models. Due to a shortage of well-labeled training data, i.e. without noise and imbalanced distribution problems, learning DDMs is particularly challenging. Considerable literature has grown up around the theme of how to learn DDMs effectively [5]- [7].
One typical approach is to learn more discriminative features through rather deep neural networks and cost-sensitive discriminative functions [5], [8]- [11]. The other typical approaches are to reweight training samples according to esti-mation errors [12], [13] or gradient directions [14] (i.e. meta learning). Although these approaches can partially alleviate the noise and imbalanced distribution problem of training data, they are not consistent with the cognitive process of human beings; the difference is we learn new things based on previously learned knowledge. In addition, these approaches do not provide a way to distinguish noisy and underrepresented examples, and thus cannot fundamentally avoid noisy and biased solutions.
Intuitively, learning DDMs from easy to hard may be more reasonable because already-learned knowledge, in such a process, can be leveraged to learn DDMs. In addition to this, the noisy and underrepresented examples could be recognized by virtue of already-learned knowledge. Hence, we resort to a gradual learning strategy i.e., self-paced learning [15]- [18]. So far, however, there are few studies on learning DDMs in a self-paced manner. Then, a natural question arises: can SPL guide DDMs to achieve more robust and less biased solutions?
An underlying problem in SPL, which is firstly shown by this study, is that it assumes the distribution of training data is balanced, and thus the bias of solutions may be further exacerbated when such an assumption fails. The reason is that the underrepresented examples, due to large prediction error, would not be selected for training at early paces, thereby incurring unfair treatment. This means existing SPL methods have fairness issue.
To address the fairness issue in existing SPL methods as well as answer the above questions, this paper proposes a new self-paced learning method for learning DDMs. It tackles the fundamental ranking and selection problems in SPL from a new perspective: fairness. On one hand, our new method keeps SPL's advantage in robustness. On the other hand, it prevents seriously biased solutions induced by SPL. The major contributions of this work include: •   concerning some sensitive attributes. • For verification, we validate SPUDRFs on three computer vision tasks: (i) facial age estimation, (ii) head pose estimation, and (iii) gaze estimation. Extensive experimental results on the Morph II [19], FG-NET [20], BIWI [21] , BU3DFE [22] and MPIIGaze [23] datasets demonstrate the efficacy of our proposed new self-paced method. On all the above tasks, SPUDRFs almost achieve state-ofthe-art performance.
A preliminary version of this work was published in [24]. This paper significantly improves [24] in the following aspects. (i) We extend our model to incorporate capped likelihood, which could further promote robustness. The robust SPUDRFs can recognize and exclude examples with labeling noise, whereas the original is unable to do so, and only places more emphasis on reliable examples. (ii) We extend our model to combine with various weighting schemes, whereas [24] merely adopts a mixture weighting scheme. (iii) We define the fairness metric for regression in this work and show obvious fairness improvement of SPUDRFs against the original SP-DRFs. (iv) We also evaluate SPUDRFs on a new computer vision task, i.e., gaze estimation, and demonstrate its validity on the MPIIGaze dataset. (v) Both robustness and fairness evaluation results are added, along with more comprehensive discussions.
This paper is organized as follows. Sec. II introduces related work on SPL, and DDMs for facial age estimation, head pose estimation and gaze estimation. Sec. III details our proposed SPUDRFs. Sec. IV introduces robust SPUDRFs. Sec. V defines a fairness metric for regression problem. Sec. VI exhibits and analyzes the experimental results on three computer vision tasks and five related datasets. Sec. VII discusses the strengths and potential weaknesses of this work, followed by Sec. VIII concluding this paper with perspectives.
II. RELATED WORK This section reviews SPL methods and DDMs-based facial age estimation, head pose estimation and gaze estimation.

A. Self-Paced Learning
Self-paced learning, as a gradual learning method, builds on the intuition that, rather than considering all training samples simultaneously, the learning should be presented with the training data from easy to difficult [15], [25]. To date, a great deal of study on self-paced learning (SPL) has been undertaken, mostly about learning conventional discriminative models, e.g. SVM, logistic regressor. There also exists some literature in which SPL is employed for clustering problems [26]- [28].
Learning conventional discriminative models in a self-paced manner exhibits superiority over traditional methods. In [15], Kumar et al. proposed the original SPL framework in a regime that suggests processing the samples in order of an easy to hard order. In [16], Zhao et al. generalized the hard weighting scheme in SPL to a soft weighting scheme, which promoted discrimination accuracy. In [17], Ma et al. proposed self-paced co-training, where SPL is applied for multi-view or multimodality problems. In [29], Ren et al. introduced capped-norm into the objective function of SPL, so as to further exclude the interference of noisy examples. In fact, the above work can be cast as incorporating SPL with conventional classifiers, e.g., SVM, logistic regressors, etc.. To incorporate SPL with regression models, in [30], Han et al. learned a mixture of regressors in a self-paced manner, and added an 1,2 norm regularizer to avoid poorly conditioned linear sub-regressors.
In computer vision community, recently some researchers have realized that learning DDMs in a self-paced manner may lead to more robust solutions. One of the few studies is [18], where Li et al. sought to enhance the learning robustness of CNNs by SPL. However, they omitted one important problem in learning discriminative models: the imbalanced distribution of training data.
Our work is inspired by the existing studies [31], [32] which take the class diversity in sample selection of SPL into consideration. Jiang et al. [32] proposed to ensure the class diversity of samples at the early paces in self-paced training. Yang et al. [31] defined a metric called "complexity of image category" to measure the number of samples in each category, as well as jointly classifying difficulty, and used it to select samples. The two methods mentioned above find that a lack of class diversity in sample selection may result in seriously biased solutions. However, what causes this problem has not been studied. This work shows that the cause is fundamentally the unfairness of sample ranking in SPL, where underrepresented examples may often have large losses and not be selected at early paces. On the other hand, [31], [32] are only suitable for classification, but not for regression in which the output targets are continuous and high dimensional. This paper will go further in this direction, aiming to tackle the fundamental drawback in SPL-ranking unfairness-and to integrate SPL with DDMs.

B. Facial Age Estimation
Facial age estimation has been extensively studied for a decade in the computer vision community. In recent years, a large and growing body of literature [3], [8], [10], [33], [34] has been proposed for age estimation with varying degrees of success. Ordinal-based approaches [3], [10] are the bestknown and have demonstrated improved results. For example, Niu et al. proposed to estimating age through a set of sequential binary queries-each query refers to a comparison with a predefined age-to exploit the inter-relationship (ordinal information) among age labels. Furthermore, Gao et al. [8] proposed DLDL-v2 to explore the underlying age distribution patterns, so as to effectively accommodate age ambiguity. Shen et al. [33] used VGG-16 to extract facial features, and mapped the extracted features to age by deep regression forests (DRFs). Li et al. [34] proposed BridgeNet to predict age by mixing local regression results, where local regressors are learned on partitioned data space. Deng et al. [35] proposed an age classification method to deal with long-tailed distribution, where a variational margin is used to minimize the influence of head classes that misleads the prediction of tailed samples. Overall, these DDMs-based approaches have improved age estimation performance significantly. However, they plausibly ignore one problem: the interference with facial images duo to PIE (i.e. pose, illumination and expression) variation, occlusion, misalignment and so forth. More importantly, these approaches mostly ignore the fact that the training data is not distributed uniformally.

C. Head Pose Estimation
Head pose estimation has attracted vast attention from the computer vision community for a long time. Recently, an increasing amount of DDMs-based methods have been proposed for head pose estimation and have dramatically boosted estimation accuracy. Of these, Riegler et al. [36] utilized convolutional neural networks (CNNs) to extract patch features within facial images, and more precise results were achieved.
In [4], Huang et al. proposed multi-modal deep regression networks, with multi-layer perceptron (MLP) architecture, to fuse the RGB and depth images for head pose estimation. In [37], Wang et al. proposed a deep coarse-to-fine network to further boost pose estimation accuracy. In [7], Ruiz et al. used a large, synthetically expanded head pose dataset to train a rather deep multi-loss convolutional neural networks (CNNs), gaining obvious accuracy improvement. In [38], Kuhnke et al. proposed a domain adaptation method for head pose estimation where shared label spaces across different domains are assumed. Although accuracy has improved, handling noisy examples in the above models is not straightforward. Moreover, the methods mentioned above seldom deal with imbalanced training data, which may widely exist in visual problems.

D. Gaze Estimation
Appearance-based gaze estimation learns a mapping from facial or eye image to gaze. Recently, a considerable amount of DDMs based approaches have been proposed for personindependent gaze estimation. For example, in [39], Zhang et al. used spatial weights VGG-16 to encode face image to gaze direction. The spatial weights were used to flexibly suppress or enhance features from different facial regions. In the literature [40], Krafka et al. implemented a CNNs-based gaze tracker by mapping captured images from the left eye, right eye and face to gaze. The location and size of the head in the original image are provided for mapping networks. In [41], Fischer et al. adopted two backbone networks to extract features from two eye regions for gaze regression, and achieved improved robustness. Instead of predicting gaze directly, in [42], Park et al. proposed another method which maps the single eye image to an intermediate pictorial representation as additional supervision to simplify 3D gaze direction estimation. Wang et al. [43] proposed a Bayesian adversarial learning approach to address the appearance, head pose variations in gaze estimation, and overfitting problem. Xiong et al. [44] proposed mixed effect neural networks to combine global and person specific gaze estimation results together. Meanwhile, other studies show that facial images can help gaze estimation. In [45]- [47], multi-channel estimation architectures were adopted to utilize facial image and eye images jointly. These methods outperform the ones that only use eye images as input for gaze estimation. In addition to person-independent methods, there is some literature about person-specific gaze adaption methods [48]- [50], which have boosted performance dramatically when a few annotated examples are used for calibration. These methods indeed improve gaze estimation results; however, whether we should learn DDMs for gaze estimation in a gradual learning manner has not been discussed, nor has there been discussion of how to deal with imbalanced distribution of training data.

III. SELF-PACED DEEP REGRESSION FORESTS WITH CONSIDERATION ON UNDERREPRESENTED SAMPLES
This section first reviews the basic concepts in DRFs; and introduces the objective formulation for SPUDRFs, as well as variant weighting strategies and an underrepresented example augmentation method; and finally details the optimization algorithm.

A. Preliminaries
Deep regression forests (DRFs), as a deep regression model, connect deep neural netwoks (DNNs) to regression forests. We start by reviewing the basic concepts in DRFs [33]. Deep Regression Tree. DRFs usually consist of a number of deep regression trees, each of which, given input-output pairs , map extracted features through DNNs to target output by passing a regression tree. Further, a regression tree T consists of split (or decision) nodes N and leaf (or prediction) nodes L [33] (see Fig. 1). Specifically, each split node n ∈ N determines whether a sample x i goes to left or right subtree; each leaf node ∈ L represents a Gaussian distribution p (y i ) with mean µ l and variance σ 2 l -parameters of output distribution defined for each tree T . Split Node. Split node represents a split function s n (x i ; Θ) : where Θ denotes the parameters of DNNs, as in Fig. 1 , where σ(·) denotes a sigmoid function while ϕ(·) denotes an index function specifying the ϕ(n)-th element of f (x i ; Θ) in correspondence with the split node n, and f (x i ; Θ) denotes the extracted features through DNNs. An example to illustrate the sketch chart of the DRFs is shown in Fig. 1(c), where ϕ 1 and ϕ 2 are two index functions for two trees. Finally, the probability of the sample x i falling into the leaf node is given by: where [·] denotes an indicator function conditioned on the argument. In addition, L nleft and L nright correspond to the sets of leaf nodes owned by the sub-trees T nleft and T nright rooted at the left and right children n l and n r of node n. Leaf Node. For a tree T , each leaf node ∈ L defines a Gaussian distribution over output y i conditioned on µ l , σ 2 l . Since each example x i has the probability ω (x i |Θ) to reach leaf node , considering all leaf nodes, the predictive distribution over y i shall sum all leaf distribution weighted by ω (x i |Θ): where π represents the distribution parameters of all leaf nodes associated with tree T . Note that π varies along with tree T k , and thus we rewrite them in terms of π k . Forests of Regression Trees. Since a forest F consists of a number of deep regression trees {T 1 , ..., T k }, the predictive output distribution shall consider all trees: where K is the number of trees and Π = {π 1 , ...,

B. Objective
Underrepresented Examples.
Considering underrepresented examples in SPL is one of the main contributions of this work. In this work, underrepresented examples mean "minority". The underrepresented level could be measured by predictive uncertainty (i.e. entropy). In fact, underrepresented examples may incur unfair treatment in early paces due to imbalanced data distribution. Our proposed method tackles the ranking and selection problems in SPL from a new perspective: fairness.
Entropy. Given a sample x i , its predictive uncertainty is formulated by calculating the entropy of the conditional output distribution p F (y i |x i , Θ, Π): where H [·] denotes entropy, and the entropy which corresponds to the k th tree is: the more uncertain the prediction should be, i.e., the more underrepresented the sample x i is.
Because the integral of mixture of Gaussians in Eq. (5) is non-trivial, we resort to Monte Carlo sampling. However, because this has a large computational cost [51], we turn to use the lower bound of this integral to approximate the integration: Let v i denote a latent variable indicating whether the i th sample is selected (v i = 1) or not (v i = 0) for training. Our objective is to jointly maximize the log likelihood function with respect to DRFs' parameters Θ and Π, and learn the latent variables v = (v 1 , ..., v N ) T . Fig. 2 shows that the original self-paced method may miss the underrepresented examples at an early pace, resulting in ignorance.
Considering fairness, we prefer to select the underrepresented examples, which probably have higher predictive uncertainty (i.e. entropy), particularly at an early pace. Therefore, we maximize the likelihood function of DRFs coupled with the sample selection term meanwhile considering ranking fairness, where λ controls learning pace (λ ≥ 0), γ is a tradeoff between loss and uncertainty (λ ≥ 0), H i denotes the entropy of output distribution p F (y i |x i , Θ, Π). When γ decays to 0, the objective function is equivalent to the loglikelihood function with respect to DRFs' parameters Θ and Π. In Eq. (7), each sample is weighted by v i , and whether That is, the sample with high likelihood value or entropy may be selected. The optimal v * i is: where p F (y i |x i , Θ, Π) is denoted by p F i for simplicity, and we use this representation in all following parts. Iteratively increasing λ and decreasing γ, samples are dynamically involved for training DRFs, starting with easy and underrepresented examples and ending up with all samples. Thus, SPUDRFs are prone to achieve more robust and less biased solutions. One might argue that the noisy and hard examples tend to have high predictive uncertainty, with the result that they are selected in the early pace. In fact, from Eq. (8), we observe that whether a sample is selected is determined by both its predictive uncertainty and the log likelihood of being predicted correctly. The noisy and hard examples probably have relatively large loss i.e. low log likelihood log p F i , and would not be selected in the early pace.

C. Weighting Schemes
Weighting each sample according to its importance would be more reasonable in SPUDRFs. In the previous section, we adopted a hard weighting scheme in SPUDRFs, as defined in Eq. (7), where one sample selected or not is determined by v i . Such a weighting scheme appears to omit the importance of samples. SPUDRFs can easily incorporate other weighting schemes, including mixture weighting and soft weighting [52]. Mixture Weighting. Mixture weighting scheme [52] weights selected sample by its importance, i.e., 0 ≤ v i ≤ 1. The objective function under mixture weighting scheme is: where ζ is a parameter controlling the learning pace. We set ζ = 1 λ − 1 λ −1 , and λ > λ > 0 to construct a reasonable soft weighting formulation. The self-paced regularizer in Eq. (9) is convex with respect to v ∈ [0, 1]. Then, setting the partial gradient of Eq. (9) with respect to v i to be zero will lead the following equation: The optimal solution of v i is given by: If either the log likelihood or the entropy is too large, v * i equals 1. In addition, if the likelihood and entropy are both too small, v * i equals 0. Except for the above two cases, the soft weighting, i.e., the last line of Eq. (11), is adopted. Soft Weighting. A soft weighting scheme [52] weights a selected sample with respect to its output likelihood and entropy. Such a weighting scheme includes: linear weighting scheme and logarithmic weighting scheme. Linear weighting. This scheme linearly assigns weights to samples with respect to output likelihood and entropy. The objective of SPUDRFs under linear weighting scheme is formulated as: where We set the partial gradient of Eq. (12) with respect to v i to be zero, then the optimal solution for v i is: The larger the log likelihood and entropy are, the higher the weight associated with the i th sample should be. Logarithmic weighting. This scheme penalizes the output likelihood and entropy logarithmically. The objective of SPU-DRFs under a logarithmic weighting scheme can be formulated as: where

D. Underrepresented Example Augmentation
As previously explained, the SPUDRFs method places more emphasis on underrepresented examples and may achieve less biased solutions. Since the intrinsic reason for SPL's biased solutions is the ignorance of underrepresented examples, we further rebalance training data via distribution reconstruction. Specifically, we distinguish the underrepresented examples whose H i are larger than β, from regular examples at each pace and augment them. As the number of underrepresented examples increases through augmentation, the label distribution imbalance problem is alleviated.

Algorithm 1
The training process of SPUDRFs.

E. Optimization
To optimize SPUDRFs, as defined in Eq. (7), we propose a two-step alternative search strategy (ASS) algorithm: (i) For sample selection, update v with fixed Θ and Π (ii) update Θ and Π with current fixed sample weights v. In addition, with fixed v, our DRFs are learned by alternatively updating Θ and Π. In [33], the parameters Θ for split nodes (i.e. parameters for VGG) are updated through gradient descent since the loss is differentiable with respect to Θ. In comparison, the parameters Π for leaf nodes are updated by virtue of variational bounding [33] when fixing Θ.

EXAMPLES
As has already been discussed in Sec. III, SPL tends to place more emphasis on reliable examples to achieve more robust solutions. However, SPL's intrinsic selection scheme can not exclude noisy examples, especially the examples with labeling noise. To alleviate this drawback, we introduce capped-likelihood function in SPUDRFs, which can render an output likelihood with an especially small value as zero: where denotes the threshold and > 0.
The robust SPUDRFs can be optimized using the optimization method proposed in Sec. III-E. Note that, if p Fi ≤ , the capped likelihood p c Fi = 0, that is, log p c Fi = −∞. Since λ is a positive factor, maximizing Eq. (17) must yield v i = 0, meaning that the i th sample would not be selected. By adjusting , we can exclude a certain ratio of noisy examples to obtain more robust solutions.

V. FAIRNESS METRIC
In addition to accuracy, fairness should also be an essential metric to measure the performance of a regression model, particularly when the regression targets are related to sensitive attributes. For example, in age estimation, we expect our model to treat the younger and the elder fairly, i.e., not to result in large MAEs for the former but small MAEs for the latter. The notion of fairness was originally defined concerning a protected attribute such as gender, race or age. However, the term fairness has a range of potential definitions. Here, we adopt a notion of fairness that is for sensitive features [53]. Defining a fairness metric for regression is one of the contributions of this work.
The present studies [54]- [57] mostly refer to fairness constraints in regressions, for example, statistical parity or bounded group loss [54], [56], but seldom refers to fairness metrics. In this work, we define a new fairness metric for regression models. To be specific, the test dataset is divided into K subsets, each of which is denoted by D k = {x i , y i |y i ∈ G k }, and G k denotes the k th group. A fair model is expected to have the same performance on all subsets, which can be described mathematically as: denotes the expectation with respect to the loss L (·) over group D k ,ŷ is the predicted value of the model and y is the real target value. Motivated by p%−rule [57], which measure classification fairness, we evaluate fairness between two subsets D k and D l as follows: The loss expectation ratio characterizes the model's fairness on every two subsets. A small ratio means the performance on such two subsets is significantly distinct, reflecting model bias against a particular subset. On the contrary, a large ratio indicates the losses on D k and D l are similar, indicating the model is relatively fair for D k and D l . In particularly, when the ratio is equal to 1, the model satisfies Eq. (18), and D k and D l are treated fairly. Finally, the regression fairness is defined as the expectation of the fairness between any two subsets, In SPUDRFs, when we set γ>0, all samples are sorted by both likelihood and entropy. As a result, easy and underrepresented samples are selected first, which means samples from all subsets would be selected at early paces. Therefore, our model can alleviate the model's prejudice against different subsets and improve regression fairness. More extensive evaluation results can be found in Sec. VI-F.

A. Datasets and Experimental Setup
Dataset. There are five datasets used in our experiments, namely Morph II, FG-NET, BIWI, BU3DFE and MPIIGaze.   [24]. The hyper-parameters of DRFs were: tree number (5), tree depth (6), output unit number of feature learning (128). The iterations to update leaf node predictions was 20, and the number of images to update leaf node predictions was the whole training set for MPIIGaze and 50 times of batch size for other datasets. In the first pace, 50% samples were selected for training. Here, λ was set to guarantee the first 50% samples with large log p F i + γH i values were involved. λ was set to ensure that 10% ∼ 20% selected samples adopt soft weighting under a mixture weighing scheme. To promote efficiency, the samples selected in the previous pace were preserved and the rest were ranked for current sample selection. In addition, γ was initialized to be 15 for Morph II and BIWI, and 5 for FG-NET, BU-3DFE, and MPIIGaze. β was used to recognize underrepresented examples that need to be augmented twice in each pace (1000 for BIWI, 400 for BU3DFE, 2000 for MPIIGaze). The number of paces was empirically set to be 10 for Morph II, 6 for BIWI and MPIIGaze, 3 for FG-NET and BU-3DFE. Except for the first pace, an equal proportion of the rest data was gradually involved in each pace. Evaluation Details. To evaluate regression accuracy, we used the mean absolute error (MAE). MAE is defined as N i=1 |ŷ i − y i | /N , whereŷ i represents the predicted output for the i th sample, and N is the total number of images for testing. In addition, for age estimation, CS calculates the percentage of images sorted in the range of [y i − L, y i + L]:   measure regression fairness, we adopt our proposed fairness metric FAIR given in Sec.V. To calculate the above metrics, for Morph II, BIWI and BU3DFE, each dataset was divided into a training set (80%) and a testing dataset (20%). The random division was repeated 5 times, and the reported MAEs were averaged over 5 times. The leave-one-person-out protocol was used for FG-NET [33] and MPIIGaze [23], where one subject was used for testing and the left subjects for training.

B. Validity of SP-DRFs and SPUDRFs
This section validates the original SPL method as well as the new proposed SPL method for learning DDMs. In the following, SP-DRFs denotes self-paced deep regression forests, which learn DRFs using the original SPL method. Self-Paced Learning. Recall that learning DDMs in a gradual learning manner is more consistent with the cognitive process of human beings, and the noisy examples can be distinguished by virtue of learned knowledge. That means the learning can place more emphasis on "reliable" examples. To show this, Fig. 4(a) illustrates the representative face images at each learning pace of SP-DRFs on the Morph II dataset, sorted by increasing λ and decreasing γ. The two numbers below each image are the actual age (left) and predicted age (right). We observe that the training images involved in each pace become more confusing and noisy step by step, compared to images in early paces. Since the current model is initialized by the results obtained at the last pace, the updated model is adaptively calibrated by "reliable" examples rather than by confusing and noisy ones. Thus, SP-DRFs have improved MAE and CS compared to DRFs. We observe that SP-DRFs gain improvement by 0.33 in MAE (2.17 → 1.84), by 1.55% in CS (91.3% → 92.85%), as shown in Fig. 6(a).
Some representative eye images in the gradual learning sequence for MPIIGaze dataset are shown in Fig. 4(b). The images in each panel are sorted by increasing λ and decreasing γ. The two numbers below each pair of images are the actual gaze direction (left) and the predicted gaze direction (right). The same phenomena-the easy examples are prone to be selected in the early paces, while the confusing and noisy ones are prone to be selected in the later paces-can be observed. Since the updated model at each pace is adaptively calibrated by "reliable" examples rather than by confusing and noisy ones, thus, the MAE associated with each pace decreases step by step. Finally, the MAE of SP-DRFs decreases to be 4.57, whereas the MAE of DRFs is 4.62 (see Table. II). Considering Ranking Fairness. As was mentioned in Sec. III, the existing SPL methods may exacerbate the bias of solutioins. Fig. 5 visualizes the leaf node distributions of SP-DRFs in the gradual learning process. The Gaussian means µ l associated with the 160 leaf nodes, where each 32 leaf nodes are defined for 1 tree, are plotted in each sub-figures. Three paces, i.e. the 1 st , 3 rd , and 6 th pace, are randomly chosen for visualization. For clarity, only pitch and yaw angles are shown. Meanwhile, the leaf node distributions of SPUDRFs are also visualized in Fig. 5.
In Fig. 5, the comparison between SP-DRFs and SPUDRFs validates our proposed new SPL method. In SP-DRFs, because the ranking fairness is not considered, the leaf nodes (red points in the 2 rd row) are only concentrated in a small range that most samples are distributed over, thus leading to seriously biased solutions. The poor MAEs of SP-DRFs can serve as evidence for this. In contrast, because the ranking fairness is considered in SPUDRFs, the leaf nodes are distributed over a wide range that could cover underrepresented examples, thus improving performance. SPUDRFs, in the pitch and yaw directions, achieve the best performance with MAEs of 0.71 and 0.69, compared to SP-DRFs with MAEs of 1.05 and 1.23 (47.9% and 78.3% improvements).

C. Comparison with State-of-the-art Methods
We compare SPUDRFs with other state-of-the-art methods on three vision tasks: age estimation, head pose estimation and gaze estimation.
Results on Age Estimation. The comparison between our method and other baselines on the Morph II and FG-NET datasets are shown in Fig. 6(a) and Fig. 7(b). The baseline methods include: LSVR [60], RCCA [61], OHRank [62], OR-CNN [10], Ranking-CNN [3], DRFs [33], DLDL-v2 [8], and PML [35]. The results show some consistent trends. First, SPUDRFs have superior performance compared to con-ventional discriminative models, such as LSVR [60] and OHRank [62]. Second, SP-DRFs consistently outperform other DDMs. Compared to DRFs, our gains in MAE are 0.33 on Morph II and 0.12 on FG-NET, and in CS are 1.55% and 2.91%, respectively. The promotions demonstrate that learning DRFs in a self-paced manner is more reasonable. Third, SPUDRFs outperform SP-DRFs on both MAE and CS. Fig. 6(b) shows the CS curves of SPUDRFs, SP-DRFs and other baseline methods on the Morph II dataset, and Fig. 7(b) shows the CS curves of different methods on the FG-NET dataset. On both datasets above, SPUDRFs consistently outperform other DDMs. We observe, the CS of SPUDRFs reaches 93.34% at error level L = 5, significantly outperforming DRFs by 2.04% on the Morph II dataset. We also observe that SPUDRFs outperform DRFs by 4.09% in CS, on the FG-NET dataset. The CS increase clearly validates our proposed self-paced learning method.
Results on Head Pose Estimation. Recall that considering underrepresented examples in SPL is particularly crucial when the training data has an imbalanced distribution problem. Sec. VI-B has shown the effectiveness of SP-DRFs and SPUDRFs for head pose estimation tasks on the BIWI dataset. Fig. 5   ranking fairness in SPL can obtain more reasonable results.

Method
MAE↓ CS↑ IIS-LDL [63] 5.77 N/A LARR [64] 5.07 68.9% MTWGP [65] 4.83 72.3% DIF [66] 4.80 74.3% OHRank [62] 4.48 74.4% CAM [67] 4.12 73.5% DRFs [33] 2  Results on Gaze Estimation. For gaze estimation, the accuracy comparisons between our proposed SP-DRFs, SPUDRFs and other baselines on the MPIIGaze dataset are shown in Tab. II. Both the MAE and standard deviation across all persons are reported. Note that the standard deviation of RT-GENE [41], Pict-Gaze [42], and Ordinal Loss [72] are not reported because the original studies do not provide this information. As shown in Tab. II, we observe that SPUDRFs outperform all baseline methods in MAE, confirming the effectiveness of our proposed method. Because RT-GENE [41] directly maps the features extracted by VGG-16 to gaze through multiple FC layers, it has inferior performance relative to DRFs (4.62 → 4.80). For a fair comparison, we chose RT-GENE with 1 model. Further, because SPUDRFs method learn DRFs in a self-paced manner and take into account ranking fairness, it has a further gain in MAE over DRFs (4.62 → 4.45). Pict-Gaze [42] regresses an input image to an intermediate pictorial representation and then regresses the representation to the gaze direction. Ordinal Loss [72] utilizes ordinal loss with order regularization to solve the regression problem. The two methods mentioned above do not take into consideration the differences amongst samples; they train all examples simultaneously and thus have inferior MAEs. We observe that the MAE of Pict-Gaze [42] and Ordinal Loss [72] are 4.56 and 4.49 respectively, while ours is 4.45. It's noteworthy that SP-DRFs, when compared with DRFs, only promotes MAE slightly. This is probably due to the obvious distribution difference between training data and test data in the leave-one-out setting. Method MAE↓ MPIIGaze [23] 6.30±1.80 iTracker [40] 6.20±0.85 GazeNet+ [73] 5.40±0.67 MeNets [44] 4.90±0.59 RT-GENE [41] 4.80± − − Pict-Gaze [42] 4.56± − − Ordinal Loss [72]

D. Different Weighting Schemes
A potential concern for SP-DRFs and SPUDRFs is that different weighting schemes could affect the estimation performance on a variety of visual tasks. To evaluate this, we compare SP-DRFs/SPUDRFs under different weighting schemes, including hard, mixture, and soft weighting. Under all weighting schemes, λ was set as in Sec. VI-A. Under the mixture weighting scheme, λ was set to ensure that 10% selected samples adopt soft weighting for Morph II, and 20% samples for other datasets. Table. III shows the comparison results. The performances of SP-DRFs/SPUDRFs with different weighting schemes are only slightly different. We observe that the weights for a large proportion of examples are close to 1. For example, under the log weighting scheme, only 2% of examples have weights below 0.5. Under the mixture weighting scheme, the proportion of samples whose weights are 1 can be set manually. We chose to set this proportion to be 0.8 ∼ 0.9 on different tasks. The MAEs are not guaranteed to be better than other weighting schemes.

E. Robust SPUDRFs
The intuition for SPUDRFs to work better on different visual tasks is its improved robustness, i.e., emphasizing more on "reliable" examples. To further promote the robustness, we propose robust SPUDRFs in Sec. IV, which enable SPUDRFs to handle labeling noise. We added noise to labels to test the validity of robust SPUDRFs on the above datasets. Specifically, we chose 10% samples in Morph II, BIWI or MPIIGaze datasets, and added Gaussian noise N (0, 10) to their labels. To control the proportion (0% ∼ 20%) of samples whose likelihoods are capped to be 0, i.e., the portion of samples to be excluded, we set at variant values. Fig. 8 shows the MAE curves of SPUDRFs across variant capped proportions. We observe that, when no example's likelihood is capped at 0, due to the presence of noise, the corresponding MAE ais large for each dataset. As the capped proportion grows, the MAE gradually decreases. When the capped proportion changes to become 10%, the MAE in Fig. 8  may be discarded when the capped proportion becomes over 10%.

F. Fairness Improvement
This section discusses how SPUDRFs improve regression fairness on different visual tasks.
Taking into account of underrepresented examples, SPU-DRFs show improved accuracy for age, pose and gaze estimation, compared to SP-DRFs. In this section, we show how SPUDRFs further improve regression fairness. In Sec. V, we define FAIR as a regression fairness metric. To evaluate the regression fairness of DRFs, SP-DRFs, and SPUDRFs, we divided all the datasets mentioned above into different subsets and calculated FAIR on them. For Morph II, the entire data was divided into 7 groups, i.e., [10,20], [20,30], . . . , [70,80]. A similar division was conducted on the FG-NET dataset. For BIWI and BU-3DFE, the pitch, yaw, and roll directions were regarded as independent directions, and the interval for each regression group is 10 • . For MPIIGaze, the pitch and yaw directions were regarded as relevant angles. We set the group interval to be 5 • in each direction, resulting in a total of 60 groups.   Some pose estimation results of DRFs, SP-DRFs, and SPU-DRFs for BIWI are shown in Fig. 9. The estimation accuracy for different pose groups reveals some consistent trends. First, for most groups, SP-DRFs and SPUDRFs have superior MAEs over DRFs. In Sec. III-B, we discussed that these gains in fairness are due to SPL, which can guide DDMs to achieve more reasonable solutions. Second, SP-DRFs tend to have even worse MAEs for the underrepresented groups than DRFs, for example, [ ] in yaw direction. This also serves as an evidence that our proposed self-paced method alleviates the fairness problem in existing SPL methods.

VII. DISCUSSION
In the experiments above, we have evaluated SPUDRFs against other baseline methods on different visual tasks, such as facial age estimation, head pose estimation and gaze estimation (Sec. VI-C). We also evaluated the SPUDRFs method under different weighting schemes, its extension with capped likelihood formulation, and its performance improvement on fairness. On a number of tasks and datasets, SPUDRFs and SP-DRFs outperform other baseline methods. The advantage of considering ranking fairness in SPUDRFs is most obvious for BIWI. For Morph II and BU-3DFE, the performance improvement when considering ranking fairness in SPUDRFs is also observable.
Learning DDMs in a self-paced manner has some limitations. The most noticeable one is that, in a leave-one-out setting, SP-DRFs perform only slightly better than DRFs. We speculate that this is due to the data distribution difference between training set and test set. Therefore, whether SPL can improve the performance of DDMs may largely depend on the distribution divergence between training data and test data.

VIII. CONCLUSION
This paper explored how self-paced learning guided deep discriminative models (DDMs) to obtain robust and less biased solutions on different visual tasks, e.g. facial age estimation, head pose estimation, and gaze estimation. A novel self-paced method, which considers ranking fairness, was proposed. Specifically, a new ranking scheme that jointly considers loss and fairness was proposed in SPL. Such a method was combined with a typical DDM-deep regression forests (DRFs)and led to a new model, namely deep regression forests with consideration on underrepresented examples (SPUDRFs). In addition, SPUDRFs under different weighting schemes, their extension with capped likelihood formulation, and their performance improvement on fairness were discussed. Extensive experiments on three well-known computer vision tasks demonstrated the efficacy of our proposed new selfpaced method. The future work will include exploring how to incorporate such a method with other DDMs.