We propose an approach to reduce the bias of ridge regression and regularization kernel network. When applied to a single data set the new algorithms have comparable learning performance with the original ones. When applied to incremental learning with block wise streaming data the new algorithms are more efficient due to bias reduction. Both theoretical characterizations and simulation studies are used to verify the effectiveness of these new algorithms.
A class of density estimates using a superposition of kernels where the kernel parameter can depend on the nearest neighbor distances is studied by the use of simulated data. Their performance using several measures of error is superior to that of the usual Parzen estimators. A tentative solution is given to the problem of calibrating the kernel peakedness when faced with a finite sample set.  SUMMARY A major difficulty in understanding the properties of variable bandwidth methods (Breiman, Meisel & Purcell, 1977; Abramson, 1982) is that extremely lengthy and complex algebra is needed to assess the influence of bias. Indeed, the complexity is so great that it has forced investigators to use computer algebraic manipulation to determine formulae for bias. In this note we completely eliminate these algebraic obstacles by presenting a simple easy-to-use formula which gives explicitly the bias of a variable bandwidth estimator, to arbitrarily high order, in very general problems.   The bias correction to the maximum likelihood extimates of the parameters for logistic discrimination is examined under mixture and separate sampling schemes. An existing adjustment developed under mixture sampling and based on higher derivatives of the log likelihood is modified slightly for use under separate sampling. The effect of this bias correction on the sampling properties of the misclassification rates and of the estimated posterior probabilities is discussed.    SUMMARY It is shown how, in regular parametric problems, the first-order term is removed from the asymptotic bias of maximum likelihood estimates by a suitable modification of the score function. In exponential families with canonical parameterization the effect is to penalize the likelihood by the Jeffreys invariant prior. In binomial logistic models, Poisson log linear models and certain other generalized linear models, the Jeffreys prior penalty function can be imposed in standard regression software using a scheme of iterative adjustments to the data. We analyze the finite-sample behavior of three second-order bias-corrected alternatives to the maximum likelihood estimator of the parameters that index the beta distribution. The three finite-sample corrections we consider are the conventional second-order bias corrected estimator ( ., 1997), the alternative approach introduced by Firth (1993) and the bootstrap bias correction. We present numerical results comparing the performance of these estimators for thirty-six different values of the parameter vector. Our results reveal that analytical bias corrections considerably outperform numerical bias corrections obtained from bootstrapping schemes. This paper provides an expression for bias of the maximum likelihood logistic regression estimates for use with small sample sizes. This bias correction is based on an expansion of the maximum likelihood equation. Simulation results show these corrections to be highly effective in small samples.