For multivariate nonparametric regression, functional analysis-of-variance (ANOVA) modeling aims to capture the relationship between a response and covariates by decomposing the unknown function into various components, representing main effects, two-way interactions, etc. Such an approach has been pursued explicitly in smoothing spline ANOVA modeling and implicitly in various greedy methods such as MARS. We develop a new method for functional ANOVA modeling, based on doubly penalized estimation using total-variation and empirical-norm penalties, to achieve sparse selection of component functions and their knots. For this purpose, we formulate a new class of hierarchical total variations, which measures total variations at different levels including main effects and multi-way interactions, possibly after some order of differentiation. Furthermore, we derive suitable basis functions for multivariate splines such that the hierarchical total variation can be represented as a regular Lasso penalty, and hence we extend a previous backfitting algorithm to handle doubly penalized estimation for ANOVA modeling. We present extensive numerical experiments on simulations and real data to compare our method with existing methods including MARS, tree boosting, and random forest. The results are very encouraging and demonstrate considerable gains from our method in both prediction or classification accuracy and simplicity of the fitted functions.
There are various types of total variations for bivariate functions. Suppose that the design points are on a 2D lattice, that is, @math . @cite_9 studied penalized least squares with a total-variation penalty on a bivariate function @math as follows, where @math , @math , and @math and @math are as in Definition with @math and @math . In fact, @math is a special case of Definition with @math and @math , suitable for piecewise constant functions. We extend this idea and carefully construct a class of total-variation penalties for multivariate functions of three or more variables while allowing an arbitrary order of differentiation @math . Moreover, as another extension, we employ total-variation penalties on component functions in ANOVA modeling, where the design points, in general, do not constitute a lattice in any dimensions.
@cite_9
2072081687
In this paper least squares penalized regression estimates withtotal variation penalities are considered. It is shown that theseestimators are least squares splines with locally data adaptive placed knotpoints. Algorithms and asymptotic properties are discussed.