Training generative models like Generative Adversarial Network (GAN)  is challenging for noisy data. A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper. The curriculum construction is based on the centrality of underlying clusters in data points.   The data points of high centrality takes priority of being fed into generative models during training. To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality. Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models. The experiments on cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data. An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper. Deep generative models have piqued researchers' interest in the past decade. The fruitful progress has been achieved on this topic, such as auto-encoder (Hinton & Salakhutdinov, 2006) and variational auto-encoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) , generative adversarial network (GAN) (Goodfellow et al., 2014; , normalizing flow (Rezende & Mohamed, 2015; Dinh et al., 2015; Kingma & Dhariwal, 2018) , and autoregressive models (van den Oord et al., 2016b; a; . However, it is non-trivial to train a deep generative model that can converge to a proper minimum of the associated optimization. For example, GAN suffers non-stability, mode collapse, and generative distortion during training. Many insightful algorithms have been proposed to circumvent those issues, including feature engineering (Salimans et al., 2016) , various discrimination metrics (Mao et al., 2016; Berthelot et al., 2017) , distinctive gradient penalties (Gulrajani et al., 2017; Mescheder et al., 2018) , spectral normalization to discriminator (Miyato et al., 2018) , and orthogonal regularization to generator (Brock et al., 2019) . What is particularly of interest is that the breakthrough for GANs has been made with a simple technique of progressively growing neural networks of generators and discriminators from low-resolution images to high-resolution counterparts (Karras et al., 2018a) . This kind of progressive growing also helps push the state of the arts to a new level by enabling StyleGAN to produce photo-realistic and detail-sharp results (Karras et al., 2018b) , shedding new light on wide applications of GANs in solving real problems. This idea of progressive learning is actually a general manner of cognition process (Elman, 1993; Oudeyer et al., 2007) , which has been formally named curriculum learning in machine learning (Bengio et al., 2009) . The central topic of this paper is to explore a new curriculum for training deep generative models. To facilitate robust training of deep generative models with noisy data, we propose curriculum learning with clustering. The key contributions are listed as follows: • We first summarize four representative curricula for generative models, i.e. architecture (generation capacity), semantics (data content), dimension (data space), and cluster (data structure). Among these curricula, cluster curriculum is newly proposed in this paper. • Cluster curriculum is to treat data according to the centrality of each data point, which is pictorially illustrated and explained in detail. To foster large-scale learning, we devise the active set algorithm that only needs an active data subset of small fixed size for training. • The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum. The geometry pertains to counting a small sphere packed in an ellipsoid, on which is based the percolation theory we use. The research on curriculum learning is diverse. Our work focuses on curricula that are closely related to data attributes, beyond which is not the scope we concern in this paper. Cluster curriculum is proposed for robust training of generative models. The active set of cluster curriculum is devised to facilitate scalable learning. The geometric principle behind cluster curriculum is analyzed in detail as well. The experimental results on the LSUN cat dataset and CelebA face dataset demonstrate that the generative models trained with cluster curriculum is capable of learning the optimal parameters with respect to the specified quality metric such as Fréchet inception distance and sliced Wasserstein distance. Geometric analysis indicates that the optimal curricula obtained from generative models are closely related to the critical points of the associated percolation processes established in this paper. This intriguing geometric phenomenon is worth being explored deeply in terms of the theoretical connection between generative models and high-dimensional geometry. It is worth emphasizing that the meaning of model optimality refers to the global minimum of the centrality-FID curve. As we already noted, the optimality is metric-dependent. We are able to obtain the optimal model with cluster curriculum, which does not mean that the algorithm only serves to this purpose. We know that more informative data can help learn a more powerful model covering the large data diversity. Here a trade-off arises, i.e. the robustness against noise and the capacity of fitting more data. The centrality-FID curve provides a visual tool to monitor the state of model training, thus aiding us in understanding the learning process and selecting suitable models according to noisy degree of given data. For instance, we can pick the trained model close to the optimal curriculum for heavily noisy data or the one near the end of the centrality-FID curve for datasets of little noise. In fact, this may be the most common way of using cluster curriculum. In this paper, we do not investigate the cluster-curriculum learning for the multi-class case, e.g. the ImageNet dataset with BigGAN (Brock et al., 2019) . The cluster-curriculum learning of multiple classes is more complex than that we have already analyzed on the face and cat data. We leave this study for future work.
A novel cluster-based algorithm of curriculum learning is proposed to solve the robust training of generative models.
{"src": "scitldr-AIC_train", "paper_id": "BklTQCEtwH"}