Several new methods have been proposed for performing valid inference after model selection. An older method is sampling splitting: use part of the data for model selection and part for inference. In this paper we revisit sample splitting combined with the bootstrap (or the Normal approximation). We show that this leads to a simple, assumption-free approach to inference and we establish results on the accuracy of the method. In fact, we find new bounds on the accuracy of the bootstrap and the Normal approximation for general nonlinear parameters with increasing dimension which we then use to assess the accuracy of regression inference. We show that an alternative, called the image bootstrap, has higher coverage accuracy at the cost of more computation. We define new parameters that measure variable importance and that can be inferred with greater accuracy than the usual regression coefficients. There is a inference-prediction tradeoff: splitting increases the accuracy and robustness of inference but can decrease the accuracy of the predictions.
There have been several additional approaches to this problem that don't fall in any of these broad categories. While this is a larger literature than can be addressed completely here, it includes early work on model selection @cite_1 and model averaging interpretations @cite_60 ; the impossibility results of @cite_15 , @cite_26 on random @math and model misspecification; methods based on resampling or sample splitting ; stability selection ; the conformal inference approach of @cite_22 ; goodness-of-fit tests of @cite_45 ; moment-constraint-based uniform confidence sets ; @cite_28 on inference about groups of variables under general designs; @cite_29 in the instrumental variable setting; @cite_41 on post-selection inference for @math -estimators, and the knockoffs approach of @cite_62 and later @cite_49 . Although they are not directed at linear models, @cite_39 and @cite_16 address similar problems for random forests.
@cite_26 @cite_62 @cite_22 @cite_60 @cite_28 @cite_41 @cite_29 @cite_1 @cite_39 @cite_45 @cite_49 @cite_15 @cite_16
2769275910 2340356402 2071168995 2116815574 2124142297
Abstract. More than thirty years ago Halbert White inaugurated a “modelrobust” form of statistical inference based on the “sandwich estimator” of standard error. It is asymptotically correct even under “model misspecification,” that is, when models are approximations rather than generative truths. It is well-known to be “heteroskedasticity-consistent”, but it is less well-known to be “nonlinearity-consistent” as well. Nonlinearity, however, raises fundamental issues: When fitted models are approximations, conditioning on the regressor is no longer permitted because the ancillarity argument that justifies it breaks down. Two effects occur: (1) parameters become dependent on the regressor distribution; (2) the sampling variability of parameter estimates no longer derives from the conditional distribution of the response alone. Additional sampling variability arises when the nonlinearity conspires with the randomness of the regressors to generate a 1 √ N contribution to standard errors. Asymptotically, standard errors from “model-trusting” fixedregressor theories can deviate from those of “model-robust” randomregressor theories by arbitrary magnitudes. In the case of linear models, a test will be proposed for comparing the two types of standard errors. We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conformal inference and split conformal inference, along with a related jackknife method. These methods offer different tradeoffs between statistical accuracy (length of resulting prediction intervals) and computational efficiency. As extensions, we develop a method for constructing valid in-sample prediction intervals called rank-one-out conformal inference, which has essentially the same computational efficiency as split conformal inference. We also describe an extension of our procedures for producing prediction bands with locally varying length, in order to adapt to heteroskedascity in the data. Finally, we propose a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference. Accompanying this paper is an R package conformalInference that implements all of the proposals we have introduced. In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package. Let Z = (Y , X 1, ..., X i m), i = 1, ..., n, be i.i.d. random vectors, Z ∼ F, F ∈ F . It is desired to predict Y by βjXj , where (β1, ..., βm) ∈ B ⊆ R, under a prediction loss. Suppose that m = n, α > 1, i.e., there are many more explanatory variables than observations. We consider sets B restricted by the maximal number of non-zero coefficients of their members, or by their l1 radius. We study the following asymptotic question: How ‘large’ may the set B be, so that it is still possible to select empirically a predictor whose risk under F is close to that of the best predictor in the set. Sharp bounds for orders of magnitudes are given under various assumptions on F . Algorithmic complexity of the ensuing procedures is also studied. The main message of this paper and the implications of the above derived orders are that under various sparsity assumptions on the optimal predictor there is “asymptotically no harm” in introducing many more explanatory variables than observations. Furthermore, such practice can be beneficial in comparison with a procedure that screens in advance a small subset of explanatory variables. Another main result is that ’Lasso’-type procedures, i.e., optimization under l1 constraint, could be efficient in finding optimal sparse predictors in high dimensions. Running head: Persistence and Predictor Selection. Summary. Stability selection was recently introduced by Meinshausen and Buhlmann as a very general technique designed to improve the performance of a variable selection algorithm. It is based on aggregating the results of applying a selection procedure to subsamples of the data. We introduce a variant, called complementary pairs stability selection, and derive bounds both on the expected number of variables included by complementary pairs stability selection that have low selection probability under the original procedure, and on the expected number of high selection probability variables that are excluded. These results require no (e.g. exchangeability) assumptions on the underlying model or on the quality of the original selection procedure. Under reasonable shape restrictions, the bounds can be further tightened, yielding improved error control, and therefore increasing the applicability of the methodology. We consider the problem of estimating the conditional distribution of a post-model-selection estimator where the conditioning is on the selected model. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion like AIC or by a hypothesis testing procedure) and second estimating the parameters in the selected model (e.g., by least-squares or maximum likelihood), all based on the same data set. We show that it is impossible to estimate this distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (local) minimax lower bounds on the performance of estimators for this distribution. Similar impossibility results are also obtained for the conditional distribution of linear functions (e.g., predictors) of the post-model-selection estimator.