Self-paced learning (SPL) is a recently raised methodology designed through simulating the learning principle of humans animals. A variety of SPL realization schemes have been designed for different computer vision and pattern recognition tasks, and empirically substantiated to be effective in these applications. However, the investigation on its theoretical insight is still a blank. To this issue, this study attempts to provide some new theoretical understanding under the SPL scheme. Specifically, we prove that the solving strategy on SPL accords with a majorization minimization algorithm implemented on a latent objective function. Furthermore, we find that the loss function contained in this latent objective has a similar configuration with non-convex regularized penalty (NSPR) known in statistics and machine learning. Such connection inspires us discovering more intrinsic relationship between SPL regimes and NSPR forms, like SCAD, LOG and EXP. The robustness insight under SPL can then be finely explained. We also analyze the capability of SPL on its easy loss prior embedding property, and provide an insightful interpretation to the effectiveness mechanism under previous SPL variations. Besides, we design a group-partial-order loss prior, which is especially useful to weakly labeled large-scale data processing tasks. Through applying SPL with this loss prior to the FCVID dataset, which is currently one of the biggest manually annotated video dataset, our method achieves state-of-the-art performance beyond previous methods, which further helps supports the proposed theoretical arguments.
Instead of using the heuristic strategies, @cite_15 formulated the key principle of CL as a concise SPL model. Formally, given a training dataset @math , in which @math and @math denote the @math observed sample and its label, respectively, @math denotes the loss function which calculates the cost between the ground truth label @math and the estimated one @math , and @math represents the model parameter in decision function @math . The SPL model includes a weighted loss term on all samples and a general self-paced regularizer imposed on sample weights, expressed as:
@cite_15
2131775048 2129131372 2256388387 2104274980
We observe a training set Q composed of l labeled samples (X sub 1 , spl theta sub 1 ),...,(X sub l , spl theta sub l ) and u unlabeled samples X sub 1 ',...,X sub u ' . The labels spl theta sub i are independent random variables satisfying Pr spl theta sub i =1 = spl eta , Pr spl theta sub i =2 =1- spl eta . The labeled observations X sub i are independently distributed with conditional density f sub spl theta i ( spl middot ) given spl theta sub i . Let (X sub 0 , spl theta sub 0 ) be a new sample, independently distributed as the samples in the training set. We observe X sub 0 and we wish to infer the classification spl theta sub 0 . In this paper we first assume that the distributions f sub 1 ( spl middot ) and f sub 2 ( spl middot ) are given and that the mixing parameter is unknown. We show that the relative value of labeled and unlabeled samples in reducing the risk of optimal classifiers is the ratio of the Fisher informations they carry about the parameter spl eta . We then assume that two densities g sub 1 ( spl middot ) and g sub 2 ( spl middot ) are given, but we do not know whether g sub 1 ( spl middot )=f sub 1 ( spl middot ) and g sub 2 ( spl middot )=f sub 2 ( spl middot ) or if the opposite holds, nor do we know spl eta . Thus the learning problem consists of both estimating the optimum partition of the observation space and assigning the classifications to the decision regions. Here, we show that labeled samples are necessary to construct a classification rule and that they are exponentially more valuable than unlabeled samples. This paper considers a natural error correcting problem with real valued input output. We wish to recover an input vector f spl isin R sup n from corrupted measurements y=Af+e. Here, A is an m by n (coding) matrix and e is an arbitrary and unknown vector of errors. Is it possible to recover f exactly from the data y? We prove that under suitable conditions on the coding matrix A, the input f is the unique solution to the spl lscr sub 1 -minimization problem ( spl par x spl par sub spl lscr 1 := spl Sigma sub i |x sub i |) min(g spl isin R sup n ) spl par y - Ag spl par sub spl lscr 1 provided that the support of the vector of errors is not too large, spl par e spl par sub spl lscr 0 :=| i:e sub i spl ne 0 | spl les spl rho spl middot m for some spl rho >0. In short, f can be recovered exactly by solving a simple convex optimization problem (which one can recast as a linear program). In addition, numerical experiments suggest that this recovery procedure works unreasonably well; f is recovered exactly even in situations where a significant fraction of the output is corrupted. This work is related to the problem of finding sparse solutions to vastly underdetermined systems of linear equations. There are also significant connections with the problem of recovering signals from highly incomplete measurements. In fact, the results introduced in this paper improve on our earlier work. Finally, underlying the success of spl lscr sub 1 is a crucial property we call the uniform uncertainty principle that we shall describe in detail. Curriculum learning (CL) or self-paced learning (SPL) represents a recently proposed learning regime inspired by the learning process of humans and animals that gradually proceeds from easy to more complex samples in training. The two methods share a similar conceptual learning paradigm, but differ in specific learning schemes. In CL, the curriculum is predetermined by prior knowledge, and remain fixed thereafter. Therefore, this type of method heavily relies on the quality of prior knowledge while ignoring feedback about the learner. In SPL, the curriculum is dynamically determined to adjust to the learning pace of the leaner. However, SPL is unable to deal with prior knowledge, rendering it prone to overfitting. In this paper, we discover the missing link between CL and SPL, and propose a unified framework named self-paced curriculum leaning (SPCL). SPCL is formulated as a concise optimization problem that takes into account both prior knowledge known before training and the learning progress during training. In comparison to human education, SPCL is analogous to "instructor-student-collaborative" learning mode, as opposed to "instructor-driven" in CL or "student-driven" in SPL. Empirically, we show that the advantage of SPCL on two tasks. A sequence of binary ( spl plusmn 1) random variables is generated by sampling a hard-limited version of a bandlimited process at n times the Nyquist rate. The information rate spl Iscr carried by these binary samples is investigated. It is shown by constructing a specific nonstationary, bounded, bandlimited process (the real zeros of which are independent and identically distributed, isolated, and lying in different Nyquist intervals) that spl Iscr =log sub 2 (n+1) bits per Nyquist interval is achievable. A more complicated construction in which L distinct zeros are placed in L consecutive Nyquist intervals yields achievable rates that approach (for L spl rarr spl infin ) spl Iscr arbitrarily closely, where spl Iscr =log sub 2 n + (n-1)log sub 2 [n (n-1)], n spl ges 2 (and spl Iscr =1 for n=1 and L=1). By exploiting the constraints imposed on the autocorrelation function of a stationary sign (bilevel) process with a given average transition rate, the latter expression is shown also to yield an upper bound on the achievable values of spl Iscr . The logarithmic behavior with n (n spl Gt 1) is due to the high correlation between the oversampled binary samples, and it is established that this trend is also achievable with stationary sign processes. This model may be used to gain insight into the effect of finite resolution on the information (in Shannon's sense) conveyed by the sign of a bandlimited process, and also to assess the limiting performance of certain oversampling-based communication systems. >