We introduce a Gaussian process model of functions which are additive. An additive function is one which decomposes into a sum of low-dimensional functions, each depending on only a subset of the input variables. Additive GPs generalize both Generalized Additive Models, and the standard GP models which use squared-exponential kernels. Hyperparameter learning in this model can be seen as Bayesian Hierarchical Kernel Learning (HKL). We introduce an expressive but tractable parameterization of the kernel function, which allows efficient evaluation of all input interaction terms, whose number is exponential in the input dimension. The additional structure discoverable by this model results in increased interpretability, as well as state-of-the-art predictive power in regression tasks.
A closely related procedure from the statistics literature is smoothing-splines ANOVA (SS-ANOVA) @cite_7 . An SS-ANOVA model is estimated as a weighted sum of splines along each dimension, plus a sum of splines over all pairs of dimensions, all triplets, etc, with each individual interaction term having a separate weighting parameter. Because the number of terms to consider grows exponentially in the order, in practice, only terms of first and second order are usually considered. Learning in SS-ANOVA is usually done via penalized-maximum likelihood with a fixed sparsity hyperparameter.
@cite_7
2963601606 2131775048 2885062394 2078214343
In many modern machine learning applications, the outcome is expensive or time consuming to collect whereas the predictor information is easy to obtain. Semiâ€ supervised (SS) learning aims at utilizing large amounts of â€˜unlabelledâ€™ data along with small amounts of â€˜labelledâ€™ data to improve the efficiency of a classical supervised approach. Though numerous SS learning classification and prediction procedures have been proposed in recent years, no methods currently exist to evaluate the prediction performance of a working regression model. In the context of developing phenotyping algorithms derived from electronic medical records, we present an efficient twoâ€ step estimation procedure for evaluating a binary classifier based on various prediction performance measures in the SS setting. In step I, the labelled data are used to obtain a nonâ€ parametrically calibrated estimate of the conditional risk function. In step II, SS estimates of the prediction accuracy parameters are constructed based on the estimated conditional risk function and the unlabelled data. We demonstrate that, under mild regularity conditions, the estimators proposed are consistent and asymptotically normal. Importantly, the asymptotic variance of the SS estimators is always smaller than that of the supervised counterparts under correct model specification. We also correct for potential overfitting bias in the SS estimators in finite samples with crossâ€ validation and we develop a perturbation resampling procedure to approximate their distributions. Our proposals are evaluated through extensive simulation studies and illustrated with two real electronic medical record studies aiming to develop phenotyping algorithms for rheumatoid arthritis and multiple sclerosis. We observe a training set Q composed of l labeled samples (X sub 1 , spl theta sub 1 ),...,(X sub l , spl theta sub l ) and u unlabeled samples X sub 1 ',...,X sub u ' . The labels spl theta sub i are independent random variables satisfying Pr spl theta sub i =1 = spl eta , Pr spl theta sub i =2 =1- spl eta . The labeled observations X sub i are independently distributed with conditional density f sub spl theta i ( spl middot ) given spl theta sub i . Let (X sub 0 , spl theta sub 0 ) be a new sample, independently distributed as the samples in the training set. We observe X sub 0 and we wish to infer the classification spl theta sub 0 . In this paper we first assume that the distributions f sub 1 ( spl middot ) and f sub 2 ( spl middot ) are given and that the mixing parameter is unknown. We show that the relative value of labeled and unlabeled samples in reducing the risk of optimal classifiers is the ratio of the Fisher informations they carry about the parameter spl eta . We then assume that two densities g sub 1 ( spl middot ) and g sub 2 ( spl middot ) are given, but we do not know whether g sub 1 ( spl middot )=f sub 1 ( spl middot ) and g sub 2 ( spl middot )=f sub 2 ( spl middot ) or if the opposite holds, nor do we know spl eta . Thus the learning problem consists of both estimating the optimum partition of the observation space and assigning the classifications to the decision regions. Here, we show that labeled samples are necessary to construct a classification rule and that they are exponentially more valuable than unlabeled samples. Multi-layer neural networks have lead to remarkable performance on many kinds of benchmark tasks in text, speech and image processing. Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification. One approach to these estimation and related problems (local minima, colinearity, feature discovery etc.) is called Dropout (Hinton, et al 2012, 2016). The Dropout algorithm removes hidden units according to a Bernoulli random variable with probability @math prior to each update, creating random "shocks" to the network that are averaged over updates. In this paper we will show that Dropout is a special case of a more general model published originally in 1990 called the Stochastic Delta Rule, or SDR (Hanson, 1990). SDR redefines each weight in the network as a random variable with mean @math and standard deviation @math . Each weight random variable is sampled on each forward activation, consequently creating an exponential number of potential networks with shared weights. Both parameters are updated according to prediction error, thus resulting in weight noise injections that reflect a local history of prediction error and local model averaging. SDR therefore implements a more sensitive local gradient-dependent simulated annealing per weight converging in the limit to a Bayes optimal network. Tests on standard benchmarks (CIFAR) using a modified version of DenseNet shows the SDR outperforms standard Dropout in test error by approx. @math with DenseNet-BC 250 on CIFAR-100 and approx. @math in smaller networks. We also show that SDR reaches the same accuracy that Dropout attains in 100 epochs in as few as 35 epochs. We study trend filtering, a recently proposed tool of [SIAM Rev. 51 (2009) 339-360] for nonparametric regression. The trend filtering estimate is defined as the minimizer of a penalized least squares criterion, in which the penalty term sums the absolute @math th order discrete derivatives over the input points. Perhaps not surprisingly, trend filtering estimates appear to have the structure of @math th degree spline functions, with adaptively chosen knot points (we say appear'' here as trend filtering estimates are not really functions over continuous domains, and are only defined over the discrete set of inputs). This brings to mind comparisons to other nonparametric regression tools that also produce adaptive splines; in particular, we compare trend filtering to smoothing splines, which penalize the sum of squared derivatives across input points, and to locally adaptive regression splines [Ann. Statist. 25 (1997) 387-413], which penalize the total variation of the @math th derivative. Empirically, we discover that trend filtering estimates adapt to the local level of smoothness much better than smoothing splines, and further, they exhibit a remarkable similarity to locally adaptive regression splines. We also provide theoretical support for these empirical findings; most notably, we prove that (with the right choice of tuning parameter) the trend filtering estimate converges to the true underlying function at the minimax rate for functions whose @math th derivative is of bounded variation. This is done via an asymptotic pairing of trend filtering and locally adaptive regression splines, which have already been shown to converge at the minimax rate [Ann. Statist. 25 (1997) 387-413]. At the core of this argument is a new result tying together the fitted values of two lasso problems that share the same outcome vector, but have different predictor matrices.