Can One Estimate The Unconditional Distribution of Post-Model-Selection
  Estimators?
arXiv:0704.1584v1  [math.ST]  12 Apr 2007
Can One Estimate The Unconditional Distribution
of Post-Model-Selection Estimators?
Hannes Leeb
Department of Statistics, Yale University
Benedikt M. Pötscher
Department of Statistics, University of Vienna
First version: April 2005
Revised version: February 2007
Abstract
We consider the problem of estimating the unconditional distribution of a post-model-selection esti-
mator. The notion of a post-model-selection estimator here refers to the combined procedure resulting
from first selecting a model (e.g., by a model selection criterion like AIC or by a hypothesis testing
procedure) and then estimating the parameters in the selected model (e.g., by least-squares or maximum
likelihood), all based on the same data set. We show that it is impossible to estimate the unconditional
distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator
for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (lo-
cal) minimax lower bounds on the performance of estimators for the distribution; performance is here
measured by the probability that the estimation error exceeds a given threshold. These lower bounds
are shown to approach 1/2 or even 1 in large samples, depending on the situation considered. Similar
impossibility results are also obtained for the distribution of linear functions (e.g., predictors) of the
post-model-selection estimator.
AMS Mathematics Subject Classification 2000: 62F10, 62F12, 62J05, 62J07, 62C05.
Keywords: Inference after model selection, Post-model-selection estimator, Pre-test estimator, Selection of re-
gressors, Akaike’s information criterion AIC, Thresholding, Model uncertainty, Consistency, Uniform consistency,
Lower risk bound.
Research of the first author was supported by the Max Kade Foundation and by the Austrian National Science
Foundation (FWF), Grant No. P13868-MAT. A preliminary draft of the material in this paper was already written
in 1999.
1 Introduction and Overview
In many statistical applications a data-based model selection step precedes the final parameter estimation and
inference stage. For example, the specification of the model (choice of functional form, choice of regressors,
http://arxiv.org/abs/0704.1584v1
number of lags, etc.) is often based on the data. In contrast, the traditional theory of statistical inference
is concerned with the properties of estimators and inference procedures under the central assumption of
an a priori given model. That is, it is assumed that the model is known to the researcher prior to the
statistical analysis, except for the value of the true parameter vector. As a consequence, the actual statistical
properties of estimators or inference procedures following a data-driven model selection step are not described
by the traditional theory which assumes an a priori given model; in fact, they may differ substantially
from the properties predicted by this theory, cf., e.g., Danilov and Magnus (2004), Dijkstra and Veldkamp
(1988), Pötscher (1991, Section 3.3), or Rao and Wu (2001, Section 12). Ignoring the additional uncertainty
originating from the data-driven model selection step and (inappropriately) applying traditional theory can
hence result in very misleading conclusions.
Investigations into the distributional properties of post-model-selection estimators, i.e., of estimators
constructed after a data-driven model selection step, are relatively few and of recent vintage. Sen (1979)
obtained the unconditional large-sample limit distribution of a post-model-selection estimator in an i.i.d.
maximum likelihood framework, when selection is between two competing nested models. In Pötscher (1991)
the asymptotic properties of a class of post-model-selection estimators (based on a sequence of hypothesis
tests) were studied in a rather general setting covering non-linear models, dependent processes, and more
than two competing models. In that paper, the large-sample limit distribution of the post-model-selection
estimator was derived, both unconditional as well as conditional on having chosen a correct model, not
necessarily the minimal one. See also Pötscher and Novak (1998) for further discussion and a simulation
study, and Nickl (2003) for extensions. The finite-sample distribution of a post-model-selection estimator,
both unconditional and conditional on having chosen a particular (possibly incorrect) model, was derived
in Leeb and Pötscher (2003) in a normal linear regression framework; this paper also studied asymptotic
approximations that are in a certain sense superior to the asymptotic distribution derived in Pötscher (1991).
The distributions of corresponding linear predictors constructed after model selection were studied in Leeb
(2005, 2006). Related work can also be found in Sen and Saleh (1987), Kabaila (1995), Pötscher (1995),
Ahmed and Basu (2000), Kapetanios (2001), Hjort and Claeskens (2003), Dukić and Peña (2005), and Leeb
and Pötscher (2005a). The latter paper provides a simple exposition of the problems of inference post model
selection and may serve as an entry point to the present paper.
It transpires from the papers mentioned above that the finite-sample distributions (as well as the large-
sample limit distributions) of post-model-selection estimators typically depend on the unknown model pa-
rameters, often in a complicated fashion. For inference purposes, e.g., for the construction of confidence
sets, estimators of these distributions would be desirable. Consistent estimators of these distributions can
typically be constructed quite easily, e.g., by suitably replacing unknown parameters in the large-sample
limit distributions by estimators; cf. Section 2.2.1. However, the merits of such ‘plug-in’ estimators in small
samples are questionable: It is known that the convergence of the finite-sample distributions to their large-
sample limits is typically not uniform with respect to the underlying parameters (see Appendix B below and
Corollary 5.5 in Leeb and Pötscher (2003)), and there is no reason to believe that this non-uniformity will
disappear when unknown parameters in the large-sample limit are replaced by estimators. This observation
is the main motivation for the present paper to investigate in general the performance of estimators of the
distribution of a post-model-selection estimator, where the estimators of the distribution are not necessar-
ily ‘plug-in’ estimators based on the limiting distribution. In particular, we ask whether estimators of the
distribution function of post-model-selection estimators exist that do not suffer from the non-uniformity
phenomenon mentioned above. As we show in this paper the answer in general is ‘No’. We also show that
these negative results extend to the problem of estimating the distribution of linear functions (e.g., linear
predictors) of post-model-selection estimators. Similar negative results apply also to the estimation of the
mean squared error or bias of post-model-selection estimators; cf. Remark 4.7.
To fix ideas consider for the moment the linear regression model
Y = V χ+Wψ + u (1)
where V and W , respectively, represent n× k and n× l non-stochastic regressor matrices (k ≥ 1, l ≥ 1), and
the n× 1 disturbance vector u is normally distributed with mean zero and variance-covariance matrix σ2In.
We also assume for the moment that (V : W )′(V : W )/n converges to a non-singular matrix as the sample
size n goes to infinity and that limn→∞ V
′W/n 6= 0 (for a discussion of the case where this limit is zero see
Example 1 in Section 2.2.2). Now suppose that the vector χ represents the parameters of interest, while
the parameter vector ψ and the associated regressors in W have been entered into the model only to avoid
possible misspecification. Suppose further that the necessity to include ψ or some of its components is then
checked on the basis of the data, i.e., a model selection procedure is used to determine which components
of ψ are to be retained in the model, the inclusion of χ not being disputed. The selected model is then used
to obtain the final (post-model-selection) estimator χ̃ for χ. We are now interested in the unconditional
finite-sample distribution of χ̃ (appropriately scaled and centered). Denote this k-dimensional cumulative
distribution function (cdf) by Gn,θ,σ(t). As indicated in the notation, this distribution function depends on
the true parameters θ = (χ′, ψ′)′ and σ. For the sake of definiteness of discussion assume for the moment
that the model selection procedure used here is the particular ‘general-to-specific’ procedure described at
the beginning of Section 2; we comment on other model selection procedures, including Akaike’s AIC and
thresholding procedures, below.
As mentioned above, it is not difficult to construct a consistent estimator of Gn,θ,σ(t) for any t, i.e., an
estimator Ĝn(t) satisfying
Pn,θ,σ
(∣∣∣Ĝn(t)−Gn,θ,σ(t)
∣∣∣ > δ
n→∞−→ 0 (2)
for each δ > 0 and each θ, σ; see Section 2.2.1. However, it follows from the results in Section 2.2.2 that any
estimator satisfying (2), i.e., any consistent estimator of Gn,θ,σ(t), necessarily also satisfies
||θ||<R
Pn,θ,σ
(∣∣∣Ĝn(t)−Gn,θ,σ(t)
∣∣∣ > δ
n→∞−→ 1 (3)
for suitable positive constants R and δ that do not depend on the estimator. That is, while the probability
in (2) converges to zero for every given θ by consistency, relation (3) shows that it does not do so uniformly
in θ. It follows that Ĝn(t) can never be uniformly consistent (not even when restricting consideration to
uniform consistency over all compact subsets of the parameter space). Hence, a large sample size does not
guarantee a small estimation error with high probability when estimating the distribution function of a post-
model-selection estimator. In this sense, reliably assessing the precision of post-model-selection estimators
is an intrinsically hard problem. Apart from (3), we also provide minimax lower bounds for arbitrary (not
necessarily consistent) estimators of the conditional distribution function Gn,θ,σ(t). For example, we provide
results that imply that
lim inf
Ĝn(t)
||θ||<R
Pn,θ,σ
(∣∣∣Ĝn(t)−Gn,θ,σ(t)
∣∣∣ > δ
> 0 (4)
holds for suitable positive constants R and δ, where the infimum extends over all estimators of Gn,θ,σ(t).
The results in Section 2.2.2 in fact show that the balls ||θ|| < R in (3) and (4) can be replaced by suitable
balls (not necessarily centered at the origin) shrinking at the rate n−1/2. This shows that the non-uniformity
phenomenon described in (3)-(4) is a local, rather than a global, phenomenon. In Section 2.2.2 we further
show that the non-uniformity phenomenon expressed in (3) and (4) typically also arises when the parameter
of interest is not χ, but some other linear transformation of θ = (χ′, ψ′)′. As discussed in Remark 4.3, the
results also hold for randomized estimators of the unconditional distribution function Gn,θ,σ(t). Hence no
resampling procedure whatsoever can alleviate the problem. This explains the anecdotal evidence in the
literature that resampling methods are often unsuccessful in approximating distributional properties of post-
model-selection estimators (e.g., Dijkstra and Veldkamp (1988), or Freedman, Navidi, and Peters (1988)).
See also the discussion on resampling in Section 6.
The results outlined above are presented in Section 2.2 for the particular ‘general-to-specific’ model
selection procedure described at the beginning of Section 2. Analogous results for a large class of model
selection procedures, including Akaike’s AIC and thresholding procedures, are then given in Section 3, based
on the results in Section 2.2. In fact, the non-uniformity phenomenon expressed in (3)-(4) is not specific to
the model selection procedures discussed in Sections 2 and 3 of the present paper, but will occur for most
(if not all) model selection procedures, including consistent ones; cf. Sections 5 and 6 for more discussion.
Section 5 also shows that the results are – as is to be expected – by no means limited to the linear regression
model.
We focus on the unconditional distributions of post-model-selection estimators in the present paper. One
can, however, also envisage a situation where one is more interested in the conditional distribution given
the outcome of the model selection procedure. In line with the literature on conditional inference (see,
e.g., Robinson (1979) or Lehmann and Casella (1998, p. 421)), one may argue that, given the outcome
of the model selection step, the relevant object of interest is the conditional rather than the unconditional
distribution of the post-model-selection estimator. In this case similar results can be obtained and are
reported in Leeb and Pötscher (2006b). We note that on a technical level the results in Leeb and Pötscher
(2006b) and in the present paper require separate treatment.
The plan of the paper is as follows: Post-model-selection estimators based on a ‘general-to-specific’ model
selection procedure are the subject of Section 2. After introducing the basic framework and some notation,
like the family of models Mp from which the ‘general-to-specific’ model selection procedure p̂ selects, as
well as the post-model-selection estimator θ̃, the unconditional cdf Gn,θ,σ(t) of (a linear function of) the
post-model-selection estimator θ̃ is discussed in Section 2.1. Consistent estimators of Gn,θ,σ(t) are given in
Section 2.2.1. The main results of the paper are contained in Section 2.2.2 and Section 3: In Section 2.2.2
we provide a detailed analysis of the non-uniformity phenomenon encountered in (3)-(4). In Section 3 the
‘impossibility’ result from Section 2.2.2 is extended to a large class of model selection procedures including
Akaike’s AIC and to selection from a non-nested collection of models. Some remarks are collected in Section
4, while Section 5 discusses extensions and the scope of the results of the paper. Conclusions are drawn
in Section 6. All proofs as well as some auxiliary results are collected into appendices. Finally a word on
notation: The Euclidean norm is denoted by ‖·‖, and λmax(E) denotes the largest eigenvalue of a symmetric
matrix E. A prime denotes transposition of a matrix. For vectors x and y the relation x ≤ y (x < y,
respectively) denotes xi ≤ yi (xi < yi, respectively) for all i. As usual, Φ denotes the standard normal
distribution function.
2 Results for Post-Model-Selection Estimators Based on a
‘General-to-Specific’ Model Selection Procedure
Consider the linear regression model
Y = Xθ + u, (5)
where X is a non-stochastic n× P matrix with rank(X) = P and u ∼ N(0, σ2In), σ2 > 0. Here n denotes
the sample size and we assume n > P ≥ 1. In addition, we assume that Q = limn→∞X ′X/n exists and is
non-singular. In this section we shall – similar as in Pötscher (1991) – consider model selection from the
collection of nested models MO ⊆ MO+1 ⊆ · · · ⊆ MP , where O is specified by the user, and where for
0 ≤ p ≤ P the model Mp is given by
(θ1, . . . , θP )
′ ∈ RP : θp+1 = · · · = θP = 0
[In Section 3 below also general non-nested families of models will be considered.] Clearly, the model Mp
corresponds to the situation where only the first p regressors in (5) are included. For the most parsimonious
model under consideration, i.e., forMO, we assume that O satisfies 0 ≤ O < P ; if O > 0, this model contains
as free parameters only those components of the parameter vector θ that are not subject to model selection.
[In the notation used in connection with (1) we then have χ = (θ1, . . . , θO)
′ and ψ = (θO+1, . . . , θP )
Furthermore, note that M0 = {(0, . . . , 0)′} and that MP = RP . We call Mp the regression model of order p.
The following notation will prove useful. For matrices B and C of the same row-dimension, the column-
wise concatenation of B and C is denoted by (B : C). If D is an m× P matrix, let D[p] denote the m× p
matrix consisting of the first p columns of D. Similarly, let D[¬p] denote the m× (P − p) matrix consisting
of the last P −p columns of D. If x is a P × 1 vector, we write in abuse of notation x[p] and x[¬p] for (x′[p])′
and (x′[¬p])′, respectively. [We shall use the above notation also in the ‘boundary’ cases p = 0 and p = P .
It will always be clear from the context how expressions containing symbols like D[0], D[¬P ], x[0], or x[¬P ]
are to be interpreted.] As usual, the i-th component of a vector x is denoted by xi, and the entry in the i-th
row and j-th column of a matrix B is denoted by Bi,j .
The restricted least-squares estimator of θ under the restriction θ[¬p] = 0, i.e., under θp+1 = · · · = θP = 0,
will be denoted by θ̃(p), 0 ≤ p ≤ P (in case p = P the restriction being void). Note that θ̃(p) is given by the
P × 1 vector
θ̃(p) =
 (X [p]
′X [p])
X [p]′Y
(0, . . . , 0)′
where the expressions θ̃(0) and θ̃(P ), respectively, are to be interpreted as the zero-vector in RP and as the
unrestricted least-squares estimator of θ. Given a parameter vector θ in RP , the order of θ (relative to the
nested sequence of models Mp) is defined as
p0(θ) = min {p : 0 ≤ p ≤ P, θ ∈Mp} .
Hence, if θ is the true parameter vector, a model Mp is a correct model if and only if p ≥ p0(θ). We stress
that p0(θ) is a property of a single parameter, and hence needs to be distinguished from the notion of the
order of the model Mp introduced earlier, which is a property of the set of parameters Mp.
A model selection procedure is now nothing else than a data-driven (measurable) rule p̂ that selects a
value from {O, . . . , P} and thus selects a model from the list of candidate modelsMO, . . . ,MP . In this section
we shall consider as an important leading case a ‘general-to-specific’ model selection procedure based on a
sequence of hypothesis tests. [Results for a larger class of model selection procedures, including Akaike’s AIC,
are provided in Section 3.] This procedure is given as follows: The sequence of hypotheses H
0 : p0(θ) < p is
tested against the alternatives H
1 : p0(θ) = p in decreasing order starting at p = P . If, for some p > O, H
is the first hypothesis in the process that is rejected, we set p̂ = p. If no rejection occurs until even HO+10
is not rejected, we set p̂ = O. Each hypothesis in this sequence is tested by a kind of t-test where the error
variance is always estimated from the overall model (but see the discussion following Theorem 3.1 in Section
3 below for other choices of estimators of the error variance). More formally, we have
p̂ = max {p : |Tp| ≥ cp, 0 ≤ p ≤ P} , (6)
with cO = 0 in order to ensure a well-defined p̂ in the range {O,O + 1, . . . , P}. For O < p ≤ P , the
critical values cp satisfy 0 < cp < ∞ and are independent of sample size (but see also Remark 4.2). The
test-statistics are given by
nθ̃p(p)
σ̂ξn,p
(0 < p ≤ P )
with the convention that T0 = 0. Furthermore,
ξn,p =
X [p]′X [p]
(0 < p ≤ P )
denotes the nonnegative square root of the p-th diagonal element of the matrix indicated, and σ̂2 is given by
σ̂2 = (n− P )−1(Y −Xθ̃(P ))′(Y −Xθ̃(P )).
Note that under the hypothesis H
0 the statistic Tp is t-distributed with n − P degrees of freedom for 0 <
p ≤ P . It is also easy to see that the so-defined model selection procedure p̂ is conservative: The probability
of selecting an incorrect model, i.e., the probability of the event {p̂ < p0(θ)}, converges to zero as the sample
size increases. In contrast, the probability of the event {p̂ = p}, for p satisfying max{p0(θ),O} ≤ p ≤ P ,
converges to a positive limit; cf., for example, Proposition 5.4 and equation (5.6) in Leeb (2006).
The post-model-selection estimator θ̃ can now be defined as follows: On the event p̂ = p, θ̃ is given by
the restricted least-squares estimator θ̃(p), i.e.,
θ̃(p)1(p̂ = p), (7)
where 1(·) denotes the indicator function of the event shown in the argument.
2.1 The Distribution of the Post-Model-Selection Estimator
We now introduce the distribution function of a linear transformation of θ̃ and summarize some of its
properties that will be needed in the subsequent development. To this end, let A be a non-stochastic k × P
matrix of rank k, 1 ≤ k ≤ P , and consider the cdf
Gn,θ,σ(t) = Pn,θ,σ
nA(θ̃ − θ) ≤ t
(t ∈ Rk). (8)
Here Pn,θ,σ(·) denotes the probability measure corresponding to a sample of size n from (5).
Depending on the choice of the matrix A, several important scenarios are covered by (8): The cdf of
n(θ̃ − θ) is obtained by setting A equal to the P × P identity matrix IP . In case O > 0, the cdf of those
components of
n(θ̃− θ) which correspond to the parameter of interest χ in (1) can be studied by setting A
to the O×P matrix (IO : 0) as we then have Aθ = (θ1, . . . , θO)′ = χ. Finally, if A 6= 0 is an 1×P vector, we
obtain the distribution of a linear predictor based on the post-model-selection estimator. See the examples
at the end of Section 2.2.2 for more discussion.
The cdf Gn,θ,σ and its properties have been analyzed in detail in Leeb and Pötscher (2003) and Leeb
(2006). To be able to access these results we need some further notation. Note that on the event p̂ = p the
expression A(θ̃ − θ) equals A(θ̃(p) − θ) in view of (7). The expected value of the restricted least-squares
estimator θ̃(p) will be denoted by ηn(p) and is given by the P × 1 vector
ηn(p) =
 θ[p] + (X [p]
′X [p])−1X [p]′X [¬p]θ[¬p]
(0, . . . , 0)′
 (9)
with the conventions that ηn(0) = (0, . . . , 0)
′ ∈ RP and that ηn(P ) = θ. Furthermore, let Φn,p denote the
cdf of
nA(θ̃(p) − ηn(p)), i.e., the cdf of
nA times the restricted least-squares estimator based on model
Mp centered at its mean. Hence, Φn,p is the cdf of a k-variate Gaussian random vector with mean zero and
variance-covariance matrix σ2A[p](X [p]′X [p]/n)−1A[p]′ in case p > 0, and it is the cdf of point-mass at zero
in Rk in case p = 0. If p > 0 and if the matrix A[p] has full row rank k, then Φn,p has a density with respect
to Lebesgue measure, and we shall denote this density by φn,p. We note that ηn(p) depends on θ and that
Φn,p depends on σ (in case p > 0), although these dependencies are not shown explicitly in the notation.
For p > 0 we introduce
bn,p = C
n (A[p](X [p]
′X [p]/n)−1A[p]′)−, (10)
ζ2n,p = ξ
n,p − C(p)
n (A[p](X [p]
′X [p]/n)−1A[p]′)−C(p)n , (11)
with ζn,p ≥ 0. Here C
n = A[p](X [p]
′X [p]/n)−1ep, where ep denotes the p-th standard basis vector in R
and B− denotes a generalized inverse of a matrix B. [Observe that ζ2n,p is invariant under the choice of the
generalized inverse. The same is not necessarily true for bn,p, but is true for bn,pz for all z in the column-
space of A[p]. Also note that (12) below depends on bn,p only through bn,pz with z in the column-space of
A[p].] We observe that the vector of covariances between Aθ̃(p) and θ̃p(p) is precisely given by σ
2n−1C
(and hence does not depend on θ). Furthermore, observe that Aθ̃(p) and θ̃p(p) are uncorrelated if and only
if ζ2n,p = ξ
n,p if and only if bn,pz = 0 for all z in the column-space of A[p]; cf. Lemma A.2 in Leeb (2005).
Finally, for a univariate Gaussian random variable N with zero mean and variance s2, s ≥ 0, we write
∆s(a, b) for P(|N− a| < b), a ∈ R∪{−∞,∞}, b ∈ R. Note that ∆s(·, ·) is symmetric around zero in its first
argument, and that ∆s(−∞, b) = ∆s(∞, b) = 0 holds. In case s = 0, N is to be interpreted as being equal
to zero, hence a 7→ ∆0(a, b) reduces to the indicator function of the interval (−b, b).
We are now in a position to present the explicit formula for Gn,θ,σ(t) derived in Leeb (2006):
Gn,θ,σ(t) = Φn,O(t−
nA(ηn(O)− θ))
q=O+1
∆σξn,q (
nηn,q(q), scqσξn,q)h(s)ds
p=O+1
nA(ηn(p)−θ)
[ ∫ ∞
(1−∆σζn,p(
nηn,p(p) + bn,pz, scpσξn,p)) (12)
q=p+1
∆σξn,q(
nηn,q(q), scqσξn,q)h(s)ds
Φn,p(dz).
In the above display, Φn,p(dz) denotes integration with respect to the measure induced by the normal cdf
Φn,p on R
k and h denotes the density of σ̂/σ, i.e., h is the density of (n−P )−1/2 times the square-root of a
chi-square distributed random variable with n−P degrees of freedom. The finite-sample distribution of the
post-model-selection estimator given in (12) is in general not normal, e.g., it can be bimodal; see Figure 2 in
Leeb and Pötscher (2005a) or Figure 1 in Leeb (2006). [An exception where (12) is normal is the somewhat
trivial case where C
n = 0, i.e., where Aθ̃(p) and θ̃p(p) are uncorrelated, for p = O + 1, . . . , P ; see Leeb
(2006, Section 3.3) for more discussion.] We note for later use that Gn,θ,σ(t) =
p=O Gn,θ,σ(t|p)πn,θ,σ(p)
where Gn,θ,σ(t|p) represents the cdf of
nA(θ̃ − θ) conditional on the event {p̂ = p} and where πn,θ,σ(p) is
the probability of this event under Pn,θ,σ. Note that πn,θ,σ(p) is always positive for O ≤ p ≤ P ; cf. Leeb
(2006), Section 3.2.
To describe the large-sample limit of Gn,θ,σ, some further notation is necessary. For p satisfying 0 < p ≤
P , partition the matrix Q = limn→∞X
′X/n as
 Q[p : p] Q[p : ¬p]
Q[¬p : p] Q[¬p : ¬p]
where Q[p : p] is a p× p matrix. Let Φ∞,p be the cdf of a k-variate Gaussian random vector with mean zero
and variance-covariance matrix σ2A[p]Q[p : p]−1A[p]′, 0 < p ≤ P , and let Φ∞,0 denote the cdf of point-mass
at zero in Rk. Note that Φ∞,p has a Lebesgue density if p > 0 and the matrix A[p] has full row rank k; in
this case, we denote the Lebesgue density of Φ∞,p by φ∞,p. Finally, for p = 1, . . . , P , define
ξ2∞,p = (Q[p : p]
−1)p,p,
ζ2∞,p = ξ
∞,p − C(p)′∞ (A[p]Q[p : p]−1A[p]′)−C(p)∞ , (13)
b∞,p = C
∞ (A[p]Q[p : p]
−1A[p]′)−,
where C
∞ = A[p]Q[p : p]
−1ep, with ep denoting the p-th standard basis vector in R
p; furthermore, take ζ∞,p
and ξ∞,p as the nonnegative square roots of ζ
∞,p and ξ
∞,p, respectively. As the notation suggests, Φ∞,p is
the large-sample limit of Φn,p, and C
∞ , ξ
∞,p, and ζ
∞,p are the limits of C
n , ξ
n,p, and ζ
n,p, respectively;
moreover, bn,pz converges to b∞,pz for each z in the column-space of A[p]. See Lemma A.2 in Leeb (2005).
The next result describes the large-sample limit of the cdf under local alternatives to θ and is taken from
Leeb (2006, Corollary 5.6). Recall that the total variation distance between two cdfs G and G∗ on Rk is
defined as ||G−G∗||TV = supE |G(E)−G∗(E)|, where the supremum is taken over all Borel sets E. Clearly,
the relation |G(t)−G∗(t)| ≤ ||G−G∗||TV holds for all t ∈ Rk. Thus, if G and G∗ are close with respect to
the total variation distance, then G(t) is close to G∗(t), uniformly in t.
Proposition 2.1 Suppose θ ∈ RP and γ ∈ RP and let σ(n) be a sequence of positive real numbers which
converges to a (finite) limit σ > 0 as n → ∞. Then the cdf Gn,θ+γ/√n,σ(n) converges to a limit G∞,θ,σ,γ in
total variation, i.e.,
∣∣∣∣Gn,θ+γ/√n,σ(n) −G∞,θ,σ,γ
n→∞−→ 0. (14)
The large-sample limit cdf G∞,θ,σ,γ(t) is given by
Φ∞,p∗(t− β
(p∗))
q=p∗+1
∆σξ∞,q (νq, cqσξ∞,q)
p=p∗+1
z≤t−β(p)
(1−∆σζ∞,p(νp + b∞,pz, cpσξ∞,p))Φ∞,p(dz)
q=p+1
∆σξ∞,q (νq, cqσξ∞,q) (15)
where p∗ = max{p0(θ),O}. Here for 0 ≤ p ≤ P
 Q[p : p]
−1Q[p : ¬p]γ[¬p]
−γ[¬p]
with the convention that β(p) = −Aγ if p = 0 and that β(p) = (0, . . . , 0)′ if p = P . Furthermore, we have
set νp = γp + (Q[p : p]
−1Q[p : ¬p]γ[¬p])p for p > 0. [Note that β(p) = limn→∞
nA(ηn(p) − θ − γ/
for p ≥ p0(θ), and that νp = limn→∞
nηn,p(p) for p > p0(θ). Here ηn(p) is defined as in (9), but with
θ + γ/
n replacing θ.]
If p∗ > 0 and if the matrix A[p∗] has full row rank k, then the Lebesgue density φ∞,p of Φ∞,p exists for
all p ≥ p∗ and hence the density of (15) exists and is given by
φ∞,p∗(t− β
(p∗))
q=p∗+1
∆σξ∞,q (νq, cqσξ∞,q)
p=p∗+1
(1−∆σζ∞,p(νp + b∞,p(t− β
(p)), cpσξ∞,p))φ∞,p(t− β
q=p+1
∆σξ∞,q(νq, cqσξ∞,q).
Like the finite-sample distribution, the limiting distribution of the post-model-selection estimator given
in (15) is in general not normal. An exception is the case where C
∞ = 0 for p > p∗ in which case G∞,θ,σ,γ
reduces to Φ∞,P ; see Remark A.6 in Appendix A. If γ = 0, we write G∞,θ,σ(t) as shorthand for G∞,θ,σ,0(t)
in the following.
2.2 Estimators of the Finite-Sample Distribution
For the purpose of inference after model selection the finite-sample distribution of the post-model-selection-
estimator is an object of particular interest. As we have seen, it depends on unknown parameters in a
complicated manner, and hence one will have to be satisfied with estimators of this cdf. As we shall see,
it is not difficult to construct consistent estimators of Gn,θ,σ(t). However, despite this consistency result,
we shall find in Section 2.2.2 that any estimator of Gn,θ,σ(t) typically performs unsatisfactory, in that the
estimation error can not become small uniformly over (subsets of) the parameter space even as sample size
goes to infinity. In particular, no uniformly consistent estimators exist, not even locally.
2.2.1 Consistent Estimators
We construct a consistent estimator of Gn,θ,σ(t) by commencing from the asymptotic distribution. Spe-
cializing to the case γ = 0 and σ(n) = σ in Proposition 2.1, the large-sample limit of Gn,θ,σ(t) is given
G∞,θ,σ(t) = Φ∞,p∗(t)
q=p∗+1
∆σξ∞,q(0, cqσξ∞,q)
p=p∗+1
(1 −∆σζ∞,p(b∞,pz, cpσξ∞,p))Φ∞,p(dz)
q=p+1
∆σξ∞,q(0, cqσξ∞,q) (16)
with p∗ = max{p0(θ),O}. Note that G∞,θ,σ(t) depends on θ only through p∗. Let Φ̂n,p denote the cdf of a k-
variate Gaussian random vector with mean zero and variance-covariance matrix σ̂2A[p](X [p]′X [p]/n)−1A[p]′,
0 < p ≤ P ; we also adopt the convention that Φ̂n,0 denotes the cdf of point-mass at zero in Rk. [We use the
same convention for Φ̂n,p in case σ̂ = 0, which is a probability zero event.] An estimator Ǧn(t) of Gn,θ,σ(t)
is now defined as follows: We first employ an auxiliary procedure p̄ that consistently estimates p0(θ) (e.g.,
p̄ could be obtained from BIC or from a ‘general-to-specific’ hypothesis testing procedure employing critical
values that go to infinity but are o(n1/2) as n → ∞). The estimator Ǧn(t) is now given by the expression
in (16) but with p∗, σ, b∞,p, ζ∞,p, ξ∞,p, and Φ∞,p replaced by max{p̄,O}, σ̂, bn,p, ζn,p, ξn,p, and Φ̂n,p,
respectively. A little reflection shows that Ǧn is again a cdf. We have the following consistency results.
Proposition 2.2 The estimator Ǧn is consistent (in the total variation distance) for Gn,θ,σ and G∞,θ,σ.
That is, for every δ > 0
Pn,θ,σ
(∣∣∣∣Ǧn(·)−Gn,θ,σ(·)
) n→∞−→ 0, (17)
Pn,θ,σ
(∣∣∣∣Ǧn(·)−G∞,θ,σ(·)
) n→∞−→ 0 (18)
for all θ ∈ RP and all σ > 0.
While the estimator constructed above on the basis of the formula for G∞,θ,σ is consistent, it can be
expected to perform poorly in finite samples since convergence of Gn,θ,σ to G∞,θ,σ is typically not uniform
in θ (cf. Appendix B), and since in case the true θ is ‘close’ to Mp0(θ)−1 the auxiliary decision procedure p̄
(although being consistent for p0(θ)) will then have difficulties making the correct decision in finite samples.
In the next section we show that this poor performance is not particular to the estimator Ǧn constructed
above, but is a genuine feature of the estimation problem under consideration.
2.2.2 Performance Limits and Impossibility Results
We now provide lower bounds for the performance of estimators of the cdf Gn,θ,σ(t) of the post-model-
selection estimator Aθ̃; that is, we give lower bounds on the worst-case probability that the estimation
error exceeds a certain threshold. These lower bounds are large, being 1 or 1/2, depending on the situation
considered; furthermore, they remain lower bounds even if one restricts attention only to certain subsets of
the parameter space that shrink at the rate n−1/2. In this sense the ‘impossibility’ results are of a local
nature. In particular, the lower bounds imply that no uniformly consistent estimator of the cdf Gn,θ,σ(t)
exists, not even locally.
In the following, the asymptotic ‘correlation’ between Aθ̃(p) and θ̃p(p) as measured by C
limn→∞ C
n will play an important rôle. [Recall that θ̃(p) denotes the least-squares estimator of θ based
on model Mp and that Aθ is the parameter vector of interest. Furthermore, the vector of covariances be-
tween Aθ̃(p) and θ̃p(p) is given by σ
2n−1C
n with C
n = A[p](X [p]
′X [p]/n)−1ep.] Note that C
∞ equals
A[p]Q[p : p]−1ep, and hence does not depend on the unknown parameters θ or σ. In the important special
case discussed in the Introduction, cf. (1), the matrix A equals the O×P matrix (IO : 0), and the condition
∞ 6= 0 reduces to the condition that the regressor corresponding to the p-th column of (V :W ) is asymp-
totically correlated with at least one of the regressors corresponding to the columns of V . See Example 1
below for more discussion.
In the result to follow we shall consider performance limits for estimators of Gn,θ,σ(t) at a fixed value of the
argument t. An estimator of Gn,θ,σ(t) is now nothing else than a real-valued random variable Γn = Γn(Y,X).
For mnemonic reasons we shall, however, use the symbol Ĝn(t) instead of Γn to denote an arbitrary estimator
of Gn,θ,σ(t). This notation should not be taken as implying that the estimator is obtained by evaluating
an estimated cdf at the argument t, or that it is a priori constrained to lie between zero and one. We shall
use this notational convention mutatis mutandis also in subsequent sections. Regarding the non-uniformity
phenomenon, we then have a dichotomy which is described in the following two results.
Theorem 2.3 Suppose that Aθ̃(q) and θ̃q(q) are asymptotically correlated, i.e., C
∞ 6= 0, for some q sat-
isfying O < q ≤ P , and let q∗ denote the largest q with this property. Then the following holds for every
θ ∈ Mq∗−1, every σ, 0 < σ < ∞, and every t ∈ Rk: There exist δ0 > 0 and ρ0, 0 < ρ0 < ∞, such that any
estimator Ĝn(t) of Gn,θ,σ(t) satisfying
Pn,θ,σ
(∣∣∣Ĝn(t)−Gn,θ,σ(t)
∣∣∣ > δ
n→∞−→ 0 (19)
for each δ > 0 (in particular, every estimator that is consistent) also satisfies
ϑ∈Mq∗
||ϑ−θ||<ρ0/
Pn,ϑ,σ
(∣∣∣Ĝn(t)−Gn,ϑ,σ(t)
∣∣∣ > δ0
n→∞−→ 1. (20)
The constants δ0 and ρ0 may be chosen in such a way that they depend only on t, Q, A, σ, and the critical
values cp for O < p ≤ P . Moreover,
lim inf
Ĝn(t)
ϑ∈Mq∗
||ϑ−θ||<ρ0/
Pn,ϑ,σ
(∣∣∣Ĝn(t)−Gn,ϑ,σ(t)
∣∣∣ > δ0
> 0 (21)
lim inf
Ĝn(t)
ϑ∈Mq∗
||ϑ−θ||<ρ0/
Pn,ϑ,σ
(∣∣∣Ĝn(t)−Gn,ϑ,σ(t)
∣∣∣ > δ
, (22)
where the infima in (21) and (22) extend over all estimators Ĝn(t) of Gn,θ,σ(t).
Remark 2.4 Assume that the conditions of the preceding theorem are satisfied. Suppose further that p⊙,
O ≤ p⊙ < q∗, is such that either p⊙ > 0 and some row of A[p⊙] equals zero, or such that p⊙ = 0. Then
there exist δ0 > 0 and 0 < ρ0 <∞ such that the left-hand side of (21) is not less than 1/2 for each θ ∈Mp⊙ .
Theorem 2.3 a fortiori implies a corresponding ‘impossibility’ result for estimation of the functionGn,θ,σ(·)
when the estimation error is measured in the total variation distance or the sup-norm; cf. also Section 5.
It remains to consider the – quite exceptional – case where the assumption of Theorem 2.3 is not satisfied,
i.e., where C
∞ = 0, for all q in the range O < q ≤ P . Under this ‘uncorrelatedness’ condition it is indeed
possible to construct an estimator of Gn,θ,σ which is uniformly consistent: It is not difficult to see that the
asymptotic distribution of Gn,θ,σ reduces to Φ∞,P under this ‘uncorrelatedness’ condition. Furthermore, the
second half of Proposition B.1 in Appendix B shows that then the convergence of Gn,θ,σ to its large-sample
limit is uniform w.r.t. θ, suggesting Φ̂n,P , an estimated version of Φ∞,P , as an estimator for Gn,θ,σ.
Proposition 2.5 Suppose that Aθ̃(q) and θ̃q(q) are asymptotically uncorrelated, i.e., C
∞ = 0, for all q
satisfying O < q ≤ P . Then
σ∗≤σ≤σ∗
Pn,θ,σ
∣∣∣Φ̂n,P −Gn,θ,σ
n→∞−→ 0 (23)
holds for each δ > 0, and for any constants σ∗ and σ
∗ satisfying 0 < σ∗ ≤ σ∗ <∞.
Inspection of the proof of Proposition 2.5 shows that (23) continues to hold if the estimator Φ̂n,P is
replaced by any of the estimators Φ̂n,p for O ≤ p ≤ P . We also note that in case O = 0 the assumption
of Proposition 2.5 is never satisfied in view of Proposition 4.4 in Leeb and Pötscher (2006b), and hence
Theorem 2.3 always applies in that case. Another consequence of Proposition 4.4 in Leeb and Pötscher
(2006b) is that – under the ‘uncorrelatedness’ assumption of Proposition 2.5 – the restricted least squares
estimators Aθ̃(q) for q ≥ O perform asymptotically as well as the unrestricted estimator Aθ̃(P ); this clearly
shows that the case covered by Proposition 2.5 is highly exceptional.
In summary we see that it is typically impossible to construct an estimator of Gn,θ,σ(t) which performs
reasonably well even asymptotically. Whenever Theorem 2.3 applies, any estimator of Gn,θ,σ(t) suffers
from a non-uniformity defect which is caused by parameters belonging to shrinking ‘tubes’ surrounding
Mq∗−1. For the sake of completeness, we remark that outside a ‘tube’ of fixed positive radius that surrounds
Mq∗−1 the non-uniformity need not be present: Let q
∗ be as in Theorem 2.3 and define the set U as
U = {θ ∈ RP : |θq∗ | ≥ r} for some fixed r > 0. Then Φ̂n,P (t) is an estimator of Gn,θ,σ(t) that is uniformly
consistent over θ ∈ U ; more generally, it can be shown that then the relation (23) holds if the supremum
over θ on the left-hand side is restricted to θ ∈ U .
We conclude this section by illustrating the above results with some important examples.
Example 1: (The distribution of χ̃) Consider the model given in (1) with χ representing the parameter
of interest. Using the general notation of Section 2, this corresponds to the case Aθ = (θ1, . . . , θO)
′ = χ
with A representing the O × P matrix (IO : 0). Here k = O > 0. The cdf Gn,θ,σ then represents the cdf
n (χ̃− χ). Assume first that limn→∞ V ′W/n 6= 0. Then C(q)∞ 6= 0 holds for some q > O. Consequently,
the ‘impossibility’ results for the estimation of Gn,θ,σ given in Theorem 2.3 always apply. Next assume
that limn→∞ V
′W/n = 0. Then C
∞ = 0 for every q > O. In this case Proposition 2.5 applies and a
uniformly consistent estimator of Gn,θ,σ indeed exists. Summarizing we note that any estimator of Gn,θ,σ
suffers from the non-uniformity phenomenon except in the special case where the columns of V and W
are asymptotically orthogonal in the sense that limn→∞ V
′W/n = 0. But this is precisely the situation
where inclusion or exclusion of the regressors in W has no effect on the distribution of the estimator χ̃
asymptotically; hence it is not surprising that also the model selection procedure does not have an effect on
the estimation of the cdf of the post-model-selection estimator χ̃. This observation may tempt one to enforce
orthogonality between the columns of V and W by either replacing the columns of V by their residuals from
the projection on the column space ofW or vice versa. However, this is not helpful for the following reasons:
In the first case one then in fact avoids model selection as all the restricted least-squares estimators for χ
under consideration (and hence also the post-model selection estimator χ̃) in the reparameterized model
coincide with the unrestricted least-squares estimator. In the second case the coefficients of the columns of
V in the reparameterized model no longer coincide with the parameter of interest χ (and again are estimated
by one and the same estimator regardless of inclusion/exclusion of columns of the transformed W -matrix).
Example 2: (The distribution of θ̃) For A equal to IP , the cdf Gn,θ,σ is the cdf of
n(θ̃ − θ). Here,
Aθ̃(q) reduces to θ̃(q), and hence Aθ̃(q) and θ̃q(q) are perfectly correlated for every q > O. Consequently,
the ‘impossibility’ result for estimation of Gn,θ,σ given in Theorem 2.3 applies. [In fact, the slightly stronger
result mentioned in Remark 2.4 always applies here.] We therefore see that estimation of the distribution of
the post-model-selection estimator of the entire parameter vector is always plagued by the non-uniformity
phenomenon.
Example 3: (The distribution of a linear predictor) Suppose A 6= 0 is a 1×P vector and one is interested
in estimating the cdf Gn,θ,σ of the linear predictor Aθ̃. Then Theorem 2.3 and the discussion following
Proposition 2.5 show that the non-uniformity phenomenon always arises in this estimation problem in case
O = 0. In case O > 0, the non-uniformity problem is generically also present, except in the degenerate case
where C
∞ = 0, for all q satisfying O < q ≤ P (in which case Proposition 4.4 in Leeb and Pötscher (2006b)
shows that the least-squares predictors from all models Mp, O ≤ p ≤ P , perform asymptotically equally
well).
3 Extensions to Other Model Selection Procedures Including AIC
In this section we show that the ‘impossibility’ result obtained in the previous section for a ‘general-to-
specific’ model selection procedure carries over to a large class of model selection procedures, including
Akaike’s widely used AIC. Again consider the linear regression model (5) with the same assumptions on the
regressors and the errors as in Section 2. Let {0, 1}P denote the set of all 0-1 sequences of length P . For
each r ∈{0, 1}P let Mr denote the set {θ ∈ RP : θi(1 − ri) = 0 for 1 ≤ i ≤ P} where ri represents the i-th
component of r. I.e., Mr describes a linear submodel with those parameters θi restricted to zero for which
ri = 0. Now let R be a user-supplied subset of {0, 1}P . We consider model selection procedures that select
from the set R, or equivalently from the set of models {Mr : r ∈ R}. Note that there is now no assumption
that the candidate models are nested (for example, if R = {0, 1}P all possible submodels are candidates for
selection). Also cases where the inclusion of a subset of regressors is undisputed on a priori grounds are
obviously covered by this framework upon suitable choice of R.
We shall assume throughout this section that R contains rfull = (1, . . . , 1) and also at least one element r∗
satisfying |r∗| = P − 1, where |r∗| represents the number of non-zero coordinates of r∗. Let r̂ be an arbitrary
model selection procedure, i.e., r̂ = r̂(Y,X) is a random variable taking its values in R. We furthermore
assume throughout this section that the model selection procedure r̂ satisfies the following mild condition:
For every r∗ ∈ R with |r∗| = P − 1 there exists a positive finite constant c (possibly depending on r∗) such
that for every θ ∈Mr∗ which has exactly P − 1 non-zero coordinates
Pn,θ,σ ({r̂ = rfull}N{|Tr∗ | ≥ c}) = lim
Pn,θ,σ ({r̂ = r∗}N{|Tr∗ | < c}) = 0 (24)
holds for every 0 < σ < ∞. Here N denotes the symmetric difference operator and Tr∗ represents the usual
t-statistic for testing the hypothesis θi(r∗) = 0 in the full model, where i(r∗) denotes the index of the unique
coordinate of r∗ that equals zero.
The above condition is quite natural for the following reason: For θ ∈ Mr∗ with exactly P − 1 non-zero
coordinates, every reasonable model selection procedure will – with probability approaching unity – decide
only betweenMr∗ andMrfull ; it is then quite natural that this decision will be based (at least asymptotically)
on the likelihood ratio between these two models, which in turn boils down to the t-statistic. As will be
shown below, condition (24) holds in particular for AIC-like procedures.
Let A be a non-stochastic k×P matrix of full row rank k, 1 ≤ k ≤ P , as in Section 2.1. We then consider
the cdf
Kn,θ,σ(t) = Pn,θ,σ
nA(θ̄ − θ) ≤ t
(t ∈ Rk) (25)
of a linear transformation of the post-model-selection estimator θ̄ obtained from the model selection procedure
r̂, i.e.,
θ̃(r)1(r̂ = r)
where the P × 1 vector θ̃(r) represents the restricted least-squares estimator obtained from model Mr, with
the convention that θ̃(r) = 0 ∈ RP in case r = (0, . . . , 0). We then obtain the following result for estimation
of Kn,θ,σ(t) at a fixed value of the argument t which parallels the corresponding ‘impossibility’ result in
Theorem 2.3.
Theorem 3.1 Let r∗ ∈ R satisfy |r∗| = P − 1, and let i(r∗) denote the index of the unique coordinate of
r∗ that equals zero; furthermore, let c be the constant in (24) corresponding to r∗. Suppose that Aθ̃(rfull)
and θ̃i(r∗)(rfull) are asymptotically correlated, i.e., AQ
i(r∗)
6= 0, where e
i(r∗)
denotes the i(r∗)-th standard
basis vector in RP . Then for every θ ∈ Mr∗ which has exactly P − 1 non-zero coordinates, for every σ,
0 < σ <∞, and for every t ∈ Rk the following holds: There exist δ0 > 0 and ρ0, 0 < ρ0 <∞, such that any
estimator K̂n(t) of Kn,θ,σ(t) satisfying
Pn,θ,σ
(∣∣∣K̂n(t)−Kn,θ,σ(t)
∣∣∣ > δ
n→∞−→ 0 (26)
for each δ > 0 (in particular, every estimator that is consistent) also satisfies
||ϑ−θ||<ρ0/
Pn,ϑ,σ
(∣∣∣K̂n(t)−Kn,ϑ,σ(t)
∣∣∣ > δ0
n→∞−→ 1 . (27)
The constants δ0 and ρ0 may be chosen in such a way that they depend only on t, Q,A, σ, and c. Moreover,
lim inf
K̂n(t)
||ϑ−θ||<ρ0/
Pn,ϑ,σ
(∣∣∣K̂n(t)−Kn,ϑ,σ(t)
∣∣∣ > δ0
> 0 (28)
lim inf
K̂n(t)
||ϑ−θ||<ρ0/
Pn,ϑ,σ
(∣∣∣K̂n(t)−Kn,ϑ,σ(t)
∣∣∣ > δ
≥ 1/2 (29)
hold, where the infima in (28) and (29) extend over all estimators K̂n(t) of Kn,θ,σ(t).
The basic condition (24) on the model selection procedure employed in the above result will certainly
hold for any hypothesis testing procedure that (i) asymptotically selects only correct models, (ii) employs a
likelihood ratio test (or an asymptotically equivalent test) for testing Mrfull versus smaller models (at least
versus the models Mr∗ with r∗ as in condition (24)), and (iii) uses a critical value for the likelihood ratio
test that converges to a finite positive constant. In particular, this applies to usual thresholding procedures
as well as to a variant of the ‘general-to-specific’ procedure discussed in Section 2 where the error variance
in the construction of the test statistic for hypothesis H
0 is estimated from the fitted model Mp rather
than from the overall model. We next verify condition (24) for AIC-like procedures. Let RSS(r) denote the
residual sum of squares from the regression employing model Mr and set
IC(r) = log (RSS(r)) + |r|Υn/n (30)
where Υn ≥ 0 denotes a sequence of real numbers satisfying limn→∞ Υn = Υ and Υ is a positive real number.
Of course, IC(r) = AIC(r) if Υn = 2. The model selection procedure r̂IC is then defined as a minimizer
(more precisely, as a measurable selection from the set of minimizers) of IC(r) over R. It is well-known that
the probability that r̂IC selects an incorrect model converges to zero. Hence, elementary calculations show
that condition (24) is satisfied for c = Υ1/2.
The analysis of post-model-selection estimators based on AIC-like model selection procedures given in this
section proceeded by bringing this case under the umbrella of the results obtained in Section 2. Verification of
condition (24) is the key that enables this approach. A complete analysis of post-model-selection estimators
based on AIC-like model selection procedures, similar to the analysis in Section 2 for the ‘general-to-specific’
model selection procedure, is certainly possible but requires a direct and detailed analysis of the distribution
of this post-model-selection estimator. [Even the mild condition that R contains rfull and also at least one
element r∗ satisfying |r∗| = P − 1 can then be relaxed in such an analysis.] We furthermore note that in the
special case where R = {rfull, r∗} and an AIC-like model selection procedure as in (30) is used, the results
in the above theorem in fact hold for all θ ∈Mr∗ .
4 Remarks and Extensions
Remark 4.1 Although not emphasized in the notation, all results in the paper also hold if the elements of
the design matrix X depend on sample size. Furthermore, all results are expressed solely in terms of the
distributions Pn,θ,σ(·) of Y , and hence they also apply if the elements of Y depend on sample size, including
the case where the random vectors Y are defined on different probability spaces for different sample sizes.
Remark 4.2 The model selection procedure considered in Section 2 is based on a sequence of tests which
use critical values cp that do not depend on sample size and satisfy 0 < cp < ∞ for O < p ≤ P . If these
critical values are allowed to depend on sample size such that they now satisfy cn,p → c∞,p as n → ∞ with
0 < c∞,p < ∞ for O < p ≤ P , the results in Leeb and Pötscher (2003) as well as in Leeb (2005, 2006)
continue to hold; see Remark 6.2(i) in Leeb and Pötscher (2003) and Remark 6.1(ii) in Leeb (2005). As a
consequence, the results in the present paper can also be extended to this case quite easily.
Remark 4.3 The ‘impossibility’ results given in Theorems 2.3 and 3.1 (as well as the variants thereof
discussed in the subsequent Remarks 4.4-4.7) also hold for the class of all randomized estimators (with
P ∗n,θ,σ replacing Pn,θ,σ in those results, where P
n,θ,σ denotes the distribution of the randomized sample).
This follows immediately from Lemma 3.6 and the attending discussion in Leeb and Pötscher (2006a).
Remark 4.4 a. Let ψn,θ,σ denote the expectation of θ̃ under Pn,θ,σ, and consider the cdf Hn,θ,σ(t) =
Pn,θ,σ(
nA(θ̃−ψn,θ,σ) ≤ t). Results for the cdf Hn,θ,σ quite similar to the results for Gn,θ,σ obtained
in the present paper can be established. A similar remark applies to the post-model-selection estimator
θ̄ considered in Section 3.
b. In Leeb (2006) also the cdf G∗n,θ,σ is analyzed, which correspond to a (typically infeasible) model
selection procedure that makes use of knowledge of σ. Results completely analogous to the ones in the
present paper can also be obtained for this cdf.
Remark 4.5 Results similar to the ones in Section 2.2.2 can also be obtained for estimation of the asymp-
totic cdf G∞,θ,σ(t) (or of the asymptotic cdfs corresponding to the variants discussed in the previous remark).
Since these results are of limited interest, we omit them. In particular, note that an ‘impossibility’ result
for estimation of G∞,θ,σ(t) per se does not imply a corresponding ‘impossibility’ result for estimation of
Gn,θ,σ(t), since Gn,θ,σ(t) does in general not converge uniformly to G∞,θ,σ(t) over the relevant subsets in
the parameter space; cf. Appendix B. [An analogous remark applies to the model selection procedures
considered in Section 3.]
Remark 4.6 Let πn,θ,σ(p) denote the model selection probability Pn,θ,σ(p̂ = p), O ≤ p ≤ P corresponding
to the model selection procedure discussed in Section 2. The finite-sample properties and the large-sample
limit behavior of these quantities are thoroughly analyzed in Leeb (2006); cf. also Leeb and Pötscher (2003).
For these model selection probabilities the following results can be established which we discuss here only
briefly:
a. The model selection probabilities πn,θ,σ(p) converge to well-defined large-sample limits which we denote
by π∞,θ,σ(p). Similar as in Proposition B.1 in Appendix B, the convergence of πn,θ,σ(p) to π∞,θ,σ(p)
is non-uniform w.r.t. θ. [For the case O = 0, this phenomenon is described in Corollary 5.6 of Leeb
and Pötscher (2003).]
b. The model selection probabilities πn,θ,σ(p) can be estimated consistently. However, uniformly consis-
tent estimation is again not possible. A similar remark applies to the large-sample limits π∞,θ,σ(p).
Remark 4.7 ‘Impossibility’ results similar to the ones given in Theorems 2.3 and 3.1 for the cdf can also be
obtained for other characteristics of the distribution of a linear function of a post-model-selection estimator
like the mean-squared error or the bias of
nAθ̃.
5 On the Scope of the Impossibility Results
The non-uniformity phenomenon described, e.g., in (20) of Theorem 2.3 is caused by a mechanism that can
informally be described as follows. Under the assumptions of that theorem, one can find an appropriate θ
and an appropriate sequence ϑn = θ + γ/
n exhibiting two crucial properties:
a. The probability measures Pn,ϑn,σ corresponding to ϑn are ‘close’ to the measures Pn,θ,σ corresponding
to θ, in the sense of contiguity. This entails that an estimator, that converges to some limit in probability
under Pn,θ,σ, converges to the same limit also under Pn,ϑn,σ.
b. For given t, the estimands Gn,ϑn,σ(t) corresponding to ϑn are ‘far away’ from the estimands Gn,θ,σ(t)
corresponding to θ, in the sense that Gn,ϑn,σ(t) and Gn,θ,σ(t) converge to different limits, i.e.,
G∞,θ,σ,0(t) is different from G∞,θ,σ,γ(t).
In view of Property a, an estimator Ĝn(t) satisfying Ĝn(t)−Gn,θ,σ(t) → 0 in probability under Pn,θ,σ, also
satisfies Ĝn(t)−Gn,θ,σ(t) → 0 in probability under Pn,ϑn,σ. In view of Property b, such an estimator Ĝn(t)
is hence ‘far away’ from the estimand Gn,ϑn,σ(t) with high probability under Pn,ϑn,σ. In other words, an
estimator that is close to Gn,θ,σ(t) under Pn,θ,σ must be far away from Gn,ϑn,σ(t) under Pn,ϑn,σ. Formalized
and refined, this argument leads to (20) and, as a consequence, to the non-existence of uniformly consistent
estimators for Gn,θ,σ(t). [There are a number of technical details in this formalization process that need
careful attention in order to obtain the results in their full strength as given in Sections 2 and 3.]
The above informal argument that derives (20) from Properties a and b can be refined and formalized in
a much more general and abstract framework, see Section 3 of Leeb and Pötscher (2006a) and the references
therein. That paper also provides a general framework for deriving results like (21) and (22) of Theorem
2.3. The mechanism leading to such lower bounds is similar to the one outlined above, where for some of
the results the concept of contiguity of the probability measures involved has to be replaced by closeness of
these measures in total variation distance. We use the results in Section 3 of Leeb and Pötscher (2006a) to
formally convert Properties a and b into the ‘impossibility’ results of the present paper; cf. Appendix C.
Verifying the aforementioned Property a in the context of the present paper is straightforward because
we consider a Gaussian linear model. What is technically more challenging and requires some work is the
verification of Property b; this is done in Appendix A inter alia and rests on results of Leeb (2002, 2005,
2006).
Two important observations on Properties a and b are in order: First, Property a is typically satisfied
in general parametric models under standard regularity conditions; e.g., it is satisfied whenever the model is
locally asymptotically normal. Second, Property b relies on limiting properties only and not on the finite-
sample structure of the underlying statistical model. Now, the limit distributions of post-model-selection
estimators in sufficiently regular parametric or semi-parametric models are typically the same as the limiting
distributions of the corresponding post-model-selection estimators in a Gaussian linear model (see, e.g., Sen
(1979), Pötscher (1991), Nickl (2003), or Hjort and Claeskens (2003)). Hence, establishing Property b for
the Gaussian linear model then typically establishes the same result for a large class of general parametric
or semi-parametric models.1 For example, Property b can be verified for a large class of pre-test estimators
in sufficiently regular parametric models by arguing as in Appendix A and using the results of Nickl (2003)
to reduce to the Gaussian linear case. Hence, the impossibility result given in Theorem 2.3 can be extended
1Some care has to be taken here. In the Gaussian linear case the finite-sample cdfs converge at every value of the argument t,
cf. Propisition 2.1. In a general parametric model, sometimes the asymptotic results (e.g., Hjort and Claeskens (2003, Theorem
4.1)) only guarantee weak convergence. Hence, to ensure convergence of the relevant cdfs at a given argument t as required
in Proberty b, additional considerations have to be employed. [This is, however, of no concern in the context discussed in the
next but one paragraph in this section.]
to more general parametric and semiparametric models with ease. The fact that we use a Gaussian linear
model for the analysis in the present paper is a matter of convenience rather than a necessity.
The non-uniformity results in Theorem 2.3 are for (conservative) ‘general-to-specific’ model selection
from a nested family of models. Theorem 3.1 extends this to more general (conservative) model selection
procedures (including AIC and related procedures) and to more general families of models. The proof of
Theorem 3.1 proceeds by reducing the problem to one where only two nested models are considered, and then
to appeal to the results of Theorem 2.3. The condition on the model selection procedures that enables this
reduction is condition (24). It is apparent from the discussion in Section 3 that this condition is satisfied for
many model selection procedures. Furthermore, for the same reasons as given in the preceding paragraph,
also Theorem 3.1 can easily be extended to sufficiently regular parametric and semi-parametric models.
The ‘impossibility’ results in the present paper are formulated for estimating Gn,θ,σ(t) for a given value
of t. Suppose that we are now asking the question whether the cdf Gn,θ,σ(·) viewed as a function can be
estimated uniformly consistently, where consistency is relative to a metric that metrizes weak convergence.2
Using a similar reasoning as above (which can again be made formal by using, e.g., Lemma 3.1 in Leeb and
Pötscher (2006a)) the key step now is to show that the function G∞,θ,σ,0(·) is different from the function
G∞,θ,σ,γ(·). Obviously, it is a much simpler problem to find a γ such that the functions G∞,θ,σ,0(·) and
G∞,θ,σ,γ(·) differ, than to find a γ such that the values G∞,θ,σ,0(t) and G∞,θ,σ,γ(t) for a given t differ.
Certainly, having solved the latter problem in Appendix A, this also provides an answer to the former. This
then immediately delivers the desired ‘impossibility’ result. [We note that in some special cases simpler
arguments than the ones used in Appendix A can be employed to solve the former problem: For example,
in case A = I the functions G∞,θ,σ,0(·) and G∞,θ,σ,γ(·) can each be shown to be convex combinations of
cdfs that are concentrated on subspaces of different dimensions. This can be exploited to establish without
much difficulty that the functions G∞,θ,σ,0(·) and G∞,θ,σ,γ(·) differ. For purpose of comparison we note that
for general A the distributions G∞,θ,σ,0 and G∞,θ,σ,γ can both be absolutely continuous w.r.t. Lebesgue
measure, not allowing one to use this simple argument.] Again the discussion in this paragraph extends to
more general parametric and semiparametric models without difficulty.
The present paper, including the discussion in this section, has focussed on conservative model selection
procedures. However, the discussion should make it clear that similar ‘impossibility’ results plague consistent
model selection. Section 2.3 in Leeb and Pötscher (2006a) in fact gives such an ‘impossibility’ result in a
simple case.
We close with the following observations. Verification of Property b, whether it is for G∞,θ,σ,0(t) and
G∞,θ,σ,γ(t) (for given t) or for G∞,θ,σ,0(·) and G∞,θ,σ,γ(·), shows that the post-model-selection estimator Aθ̃
is a so-called non-regular estimator for Aθ: Consider an estimator β̃ in a parametric model {Pn,β : β ∈ B}
where the parameter space B is an open subset of Euclidean space Rd. Suppose β̃, properly scaled and
centered, has a limit distribution under local alternatives, in the sense that
n(β̃ − (β + γ/
n)) converges
in law under Pn,β+γ/
n to a limit distribution L∞,β,γ(·) for every γ. The estimator β̃ is called regular if for
every β the limit distribution L∞,β,γ(·) does not depend on γ; cf. van der Vaart (1998, Section 8.5). Suppose
now that the model is, e.g., locally asymptotically normal (hence the contiguity property in Property a is
satisfied). The informal argument outlined at the beginning of this section (and which is formalized in
Lemma 3.1 of Leeb and Pötscher (2006a)) then in fact shows that the cdf of any non-regular estimator
2Or, in fact, any metric w.r.t. which the relevant cdfs converge.
can not be estimated uniformly consistently (where consistency is relative to any metric that metrizes weak
convergence).
6 Conclusions
Despite the fact that we have shown that consistent estimators for the distribution of a post-model-selection
estimator can be constructed with relative ease, we have also demonstrated that no estimator of this distri-
bution can have satisfactory performance (locally) uniformly in the parameter space, even asymptotically.
In particular, no (locally) uniformly consistent estimator of this distribution exists. Hence, the answer to
the question posed in the title has to be negative. The results in the present paper also cover the case of
linear functions (e.g., predictors) of the post-model-selection estimator.
We would like to stress here that resampling procedures like, e.g., the bootstrap or subsampling, do
not solve the problem at all. First note that standard bootstrap techniques will typically not even provide
consistent estimators of the finite-sample distribution of the post-model-selection estimator, as the bootstrap
can be shown to stay random in the limit (Kulperger and Ahmed (1992), Knight (1999, Example 3))3.
Basically the only way one can coerce the bootstrap into delivering a consistent estimator is to resample from
a model that has been selected by an auxiliary consistent model selection procedure. The consistent estimator
constructed in Section 2.2.1 is in fact of this form. In contrast to the standard bootstrap, subsampling will
typically deliver consistent estimators. However, the ‘impossibility’ results given in this paper apply to any
estimator (including randomized estimators) of the cdf of a post-model-selection estimator. Hence, also any
resampling based estimator suffers from the non-uniformity defects described in Theorems 2.3 and 3.1; cf.
also Remark 4.3.
The ‘impossibility’ results in Theorems 2.3 and 3.1 are derived in the framework of a normal linear
regression model (and a fortiori these results continue to hold in any model which includes the normal
linear regression model as a special case), but this is more a matter of convenience than anything else: As
discussed in Section 5, similar results can be obtained in general statistical models allowing for nonlinearity
or dependent data, e.g., as long as standard regularity conditions for maximum likelihood theory are satisfied.
The results in the present paper are derived for a large class of conservative model selection procedures
(i.e., procedures that select overparameterized models with positive probability asymptotically) including
Akaike’s AIC and typical ‘general-to-specific’ hypothesis testing procedures. For consistent model selection
procedures – like BIC or testing procedures with suitably diverging critical values cp (cf. Bauer, Pötscher, and
Hackl (1988)) – the (pointwise) asymptotic distribution is always normal. [This is elementary, cf. Lemma 1
in Pötscher (1991).] However, as discussed at length in Leeb and Pötscher (2005a), this asymptotic nor-
mality result paints a misleading picture of the finite sample distribution which can be far from a normal,
the convergence of the finite-sample distribution to the asymptotic normal distribution not being uniform.
‘Impossibility’ results similar to the ones presented here can also be obtained for post-model-selection esti-
mators based on consistent model selection procedures. These will be discussed in detail elsewhere. For a
3Brownstone (1990) claims the validity of a bootstrap procedure that is based on a conservative model selection procedure
in a linear regression model. Kilian (1998) makes a similar claim in the context of autoregressive models selected by a conser-
vative model selection procedure. Also Hansen (2003) contains such a claim for a stationary bootstrap procedure based on a
conservative model selection procedure. The above discussion intimates that these claims are at least unsubstantiated.
simple special case such an ‘impossibility’ result is given in Section 2.3 of Leeb and Pötscher (2006a).
The ‘impossibility’ of estimating the distribution of the post-model-selection estimator does not per se
preclude the possibility of conducting valid inference after model selection, a topic that deserves further
study. However, it certainly makes this a more challenging task.
A Auxiliary Lemmas
Lemma A.1 Let Z be a random vector with values in Rk and let W be a univariate standard Gaussian
random variable independent of Z. Furthermore, let C ∈ Rk and τ > 0. Then
P(Z ≤ Cx)P(|W − x| < τ ) + P(Z ≤ CW, |W − x| ≥ τ) (31)
is constant as a function of x ∈ R if and only if C = 0 or P(Z ≤ Cx) = 0 for each x ∈ R.
Proof of Lemma A.1: Suppose C = 0 holds. Using independence of Z andW it is then easy to see that
(31) reduces to P(Z ≤ 0), which is constant in x. If P(Z ≤ Cx) = 0 for every x ∈ R, then P(Z ≤ CW ) = 0,
and hence (31) is again constant, namely equal to zero.
To prove the converse, assume that (31) is constant in x ∈ R. Letting x→ ∞, we see that (31) must be
equal to P(Z ≤ CW ). This entails that
P(Z ≤ Cx)P(|W − x| < τ) = P(Z ≤ CW, |W − x| < τ )
holds for every x ∈ R. Write F (x) as shorthand for P(Z ≤ Cx), and let Φ(z) and φ(z) denote the cdf and
density of W , respectively. Then the expression in the above display can be written as
F (x)(Φ(x + τ )− Φ(x− τ)) =
∫ x+τ
F (z)φ(z)dz. (x ∈ R) (32)
We now further assume that C 6= 0 and that F (x) 6= 0 for at least one x ∈ R, and show that this leads to a
contradiction.
Consider first the case where all components of C are non-negative. Since F is not identically zero, it
is then, up to a scale factor, the cdf of a random variable on the real line. But then (32) can not hold for
all x ∈ R as shown in Example 7 in Leeb (2002) (cf. also equation (7) in that paper). The case where
all components of C are non-positive follows similarly by applying the above argument to F (−x) and upon
observing that both Φ(x+ τ )− Φ(x− τ) and φ(x) are symmetric around x = 0.
Finally, consider the case where C has at least one positive and one negative component. In this case
clearly limx→−∞ F (x) = limx→∞ F (x) = 0 holds. Since F (x) is continuous in view of (32), we see that F (x)
attains its (positive) maximum at some point x1 ∈ R. Now note that (32) with x1 replacing x can be written
as ∫ x1+τ
(F (x1)− F (z))φ(z)dz = 0.
This immediately entails that F (x) = F (x1) for each x ∈ [x1 − τ , x1 + τ ] (because F (x) is continuous and
because of the definition of x1). Repeating this argument with x1−τ replacing x1 and proceeding inductively,
we obtain that F (x) = F (x1) for each x satisfying x ≤ x1 + τ , a contradiction with limx→−∞ F (x) = 0. ✷
Lemma A.2 Let M and N be matrices of dimension k × p and k × q, respectively, such that the matrix
(M : N) has rank k (k ≥ 1, p ≥ 1, q ≥ 1). Let t ∈ Rk, and let V be a random vector with values in Rp
whose distribution assigns positive mass to every (non-empty) open subset of Rp (e.g., it possesses an almost
everywhere positive Lebesgue density). Set f(x) = P(MV ≤ t + Nx), x ∈ Rq. If one of the rows of M
consists of zeros only, then f is discontinuous at some point x0. More precisely, there exist x0 ∈ Rq, z ∈ Rq
and a constant c > 0, such that f(x0 + δz) ≥ c and f(x0 − δz) = 0 hold for every sufficiently small δ > 0.
Proof of Lemma A.2: The case where M is the zero-matrix is trivial. Otherwise, let I0 denote the
set of indices i, 1 ≤ i ≤ k, for which the i-th row of M is zero. Let (M0 : N0) denote the matrix consisting
of those rows of (M : N) whose index is in I0, and let (M1 : N1) denote the matrix consisting of the
remaining rows of (M : N). Clearly, M0 is then the zero matrix. Furthermore, note that N0 has full
row-rank. Moreover, let t0 denote the vector consisting of those components of t whose index is in I0 and
let t1 denote the vector containing the remaining components. With this notation, f(x) can be written as
P(0 ≤ t0 +N0x, M1V ≤ t1 +N1x).
For vectors µ ∈ Rp and η ∈ Rq to be specified in a moment, set t∗ = t+Mµ+Nη, and let t∗0 and t∗1 be
defined similarly to t0 and t1. Since the matrix (M : N) has full rank k, we can choose µ and η such that
t∗0 = 0 and t
1 > 0. Choose z ∈ Rq such that N0z > 0, which is possible because N0 has full row-rank. Set
x0 = η. Then for every ǫ ∈ R we have
f(x0 + ǫz) = f(η + ǫz) = P(MV ≤ t+N(η + ǫz))
= P(0 ≤ t0 +N0(η + ǫz), M1V ≤ t1 +N1(η + ǫz))
= P(0 ≤ t∗0 + ǫN0z, M1(V + µ) ≤ t∗1 + ǫN1z)
= P(0 ≤ ǫN0z, M1(V + µ) ≤ t∗1 + ǫN1z)
Since t∗1 > 0, we can find a t
1 such that 0 < t
1 < t
1 + ǫN1z holds for every ǫ with |ǫ| small enough. If now
ǫ > 0 then
f(x0 + ǫz) = P(M1(V + µ) ≤ t∗1 + ǫN1z) ≥ P(M1(V + µ) ≤ t∗∗1 ).
The r.h.s. in the above display is positive because t∗∗1 > 0 and because the distribution ofM1(V +µ) assigns
positive mass to any neighborhood of the origin, since the same is true for the distribution of V + µ and
since M1 maps neighborhoods of zero into neighborhoods of zero. Setting c = P(M1(V + µ) ≤ t∗∗1 )/2, we
have f(x0 + ǫz) ≥ c > 0 for each sufficiently small ǫ > 0. Furthermore, for ǫ < 0 we have f(x0 + ǫz) = 0,
since f(x0 + ǫz) ≤ P(0 ≤ ǫN0z) = 0 in view of N0z > 0. ✷
Lemma A.3 Let Z be a random vector with values in Rp, p ≥ 1, with a distribution that is absolutely
continuous with respect to Lebesgue measure on Rp. Let B be a k×p matrix, k ≥ 1. Then the cdf P(BZ ≤ ·)
of BZ, is discontinuous at t ∈ Rk if and only if P(BZ ≤ t) > 0 and if for some i0, 1 ≤ i0 ≤ k, the i0-th row
of B and the i0-th component of t are both zero, i.e., Bi0,· = (0, . . . , 0) and ti0 = 0.
Proof of Lemma A.3: To establish sufficiency of the above condition, let P(BZ ≤ t) > 0, ti0 = 0 and
Bi0,· = (0, . . . , 0) for some i0, 1 ≤ i0 ≤ k. Then, of course, P(Bi0,·Z = 0) = 1. For tn = t−n−1ei0 , where ei0
denotes the i0-th unit vector in R
k, we have P(BZ ≤ tn) ≤ P(Bi0,·Z ≤ tn,i0) = P(Bi0,·Z ≤ −1/n) = 0 for
every n. Consequently, P(BZ ≤ t) is discontinuous at t.
To establish necessity, we first show the following: If tn ∈ Rk is a sequence converging to t ∈ Rk as
n→ ∞, then every accumulation point of the sequence P(BZ ≤ tn) has the form
P(Bi1,·Z ≤ ti1 , . . . , Bim,·Z ≤ tim , Bim+1,·Z < tim+1 , . . . , Bik,·Z < tik) (33)
for somem, 0 ≤ m ≤ k, and for some permutation (i1, . . . , ik) of (1, . . . , k). This can be seen as follows: Let α
be an accumulation point of P(BZ ≤ tn). Then we may find a subsequence such that P(BZ ≤ tn) converges
to α along this subsequence. From this subsequence we may even extract a further subsequence along which
each component of the k × 1 vector tn converges to the corresponding component of t monotonously, that
is, either from above or from below. Without loss of generality, we may also assume that those components
which converge from below are strictly increasing. The resulting subsequence will be denoted by nj in the
sequel. Assume that the components of tnj with indices i1, . . . , im converge from above, while the components
with indices im+1, . . . , ik converge from below. Now
P(BZ ≤ tnj ) =
1(−∞,tnj,s](zs)PBZ(dz), (34)
where PBZ denotes the distribution of BZ. The integrand in (34) now converges to
l=1 1(−∞,til ](zil)
l=m+1 1(−∞,til )(zil) for all z ∈ R
k as nj → ∞. The r.h.s. of (34) converges to the
expression in (33) as nj → ∞ by the Dominated Convergence Theorem, while the l.h.s. of (34) converges to
α by construction. This establishes the claim regarding (33).
Now suppose that P(BZ ≤ t) is discontinuous at t; i.e., there exists a sequence tn converging to t as
n → ∞, such that P(BZ ≤ tn) does not converge to P(BZ ≤ t) as n → ∞. From the sequence tn we
can extract a subsequence tns along which P(BZ ≤ tns) converges to a limit different from P(BZ ≤ t) as
ns → ∞. As shown above, the limit has to be of the form (33) and m < k has to hold. Consequently, the
limit of P(BZ ≤ tns) is smaller than P(BZ ≤ t) = P(Bi,·Z ≤ ti, i = 1, . . . , k). The difference of P(BZ ≤ t)
and the limit of P(BZ ≤ tns) is positive and because of (33) can be written as
P(Bij ,·Z ≤ tij for each j = 1, . . . , k, Bij ,·Z = tij for some j = m+ 1, . . . , k) > 0.
We thus see that P(Bij0 ,·Z = tij0 ) > 0 for some j0 satisfying m + 1 ≤ j0 ≤ k. As Z is absolutely
continuous with respect to Lebesgue measure on Rp, this can only happen if Bij0 ,· = (0, . . . , 0) and tij0 = 0.
Lemma A.4 Suppose that Aθ̃(q) and θ̃q(q) are asymptotically correlated, i.e., C
∞ 6= 0, for some q satisfying
O < q ≤ P , and let q∗ denote the largest q with this property. Moreover let θ ∈ Mq∗−1, let σ satisfy
0 < σ < ∞, and let t ∈ Rk. Then G∞,θ,σ,γ(t) is non-constant as a function of γ ∈ Mq∗\Mq∗−1. More
precisely, there exist δ0 > 0 and ρ0, 0 < ρ0 <∞, such that
γ(1),γ(2)∈Mq∗ \Mq∗−1
||γ(i)||<ρ0,i=1,2
∣∣G∞,θ,σ,γ(1)(t)−G∞,θ,σ,γ(2)(t)
∣∣ > 2δ0 (35)
holds. The constants δ0 and ρ0 can be chosen in such a way that they depend only on t, Q, A, σ, and the
critical values cp for O < p ≤ P .
Lemma A.5 Suppose that Aθ̃(q) and θ̃q(q) are asymptotically correlated, i.e., C
∞ 6= 0, for some q satisfying
O < q ≤ P , and let q∗ denote the largest q with this property. Suppose further that for some p⊙ satisfying
O ≤ p⊙ < q∗ either p⊙ = 0 holds or that p⊙ > 0 and A[p⊙] has a row of zeros. Then, for every θ ∈ Mp⊙ ,
every σ, 0 < σ < ∞, and every t ∈ Rk the quantity G∞,θ,σ,γ(t) is discontinuous as a function of γ ∈ Mq∗ .
More precisely, for each s = O, . . . , p⊙, there exist vectors β∗ and γ∗ in Mq∗ and constants δ∗ > 0 and ǫ∗ > 0
such that
∣∣G∞,θ,σ,β∗+ǫγ∗(t)−G∞,θ,σ,β∗−ǫγ∗(t)
∣∣ ≥ δ∗ (36)
holds for every θ satisfying max{p0(θ),O} = s and for every ǫ with 0 < ǫ < ǫ∗. The quantities δ∗, ǫ∗, β∗,
and γ∗ can be chosen in such a way that – besides t, Q, A, σ, and the critical values cp for O < p ≤ P –
they depend on θ only through max{p0(θ),O}.
Before we prove the above lemmas, we provide a representation of G∞,θ,σ,γ(t) that will be useful in
the following: For 0 < p ≤ P define Zp =
r=1 ξ
∞ Wr, where C
∞ has been defined after (13) and
the random variables Wr are independent normally distributed with mean zero and variances σ
2ξ2∞,r; for
convenience, let Z0 denote the zero vector in R
k. Observe that Zp, p > 0, is normally distributed with
mean zero and variance-covariance matrix σ2A[p]Q[p : p]−1A[p]′ since it has been shown in the proof of
Proposition 4.4 in Leeb and Pötscher (2006b) that the asymptotic variance-covariance matrix σ2A[p]Q[p :
p]−1A[p]′ of
nAθ̃(p) can be expressed as
r=1 σ
2ξ−2∞,rC
∞ . Also the joint distribution of Zp and the
set of variables Wr, 1 ≤ r ≤ P , is normal, with the covariance vector between Zp and Wr given by σ2C(r)∞ in
case r ≤ p; otherwise Zp and Wr are independent. Define the constants νr = γr+(Q[r : r]−1Q[r : ¬r]γ[¬r])r
for 0 < r ≤ P . It is now easy to see that for p ≥ p∗ = max{p0(θ),O} the quantity β(p) defined in
Proposition 2.1 equals −
r=p+1 ξ
∞ νr. [This is seen as follows: It was noted in Proposition 2.1 that
β(p) = limn→∞
nA(ηn(p) − θ − γ/
n) for p ≥ p0(θ), when ηn(p) is defined as in (9), but with θ + γ/
replacing θ. Using the representation (20) of Leeb (2005) and taking limits, the result follows if we observe
nηn,r(r) −→ νr for r > p ≥ p0(θ).] The cdf in (15) can now be written as
Zp∗ ≤ t+
r=p∗+1
ξ−2∞,rC
q=p∗+1
P(|Wq + νq| < cqσξ∞,q)
p=p∗+1
Zp ≤ t+
r=p+1
ξ−2∞,rC
∞ νr, |Wp + νp| ≥ cpσξ∞,p
q=p+1
P(|Wq + νq| < cqσξ∞,q). (37)
That the terms corresponding to p = p∗ in (37) and (15) agree is obvious. Furthermore, for each p > p∗
the terms under the product sign in (37) and (15) coincide by definition of the function ∆s(a, b). It is also
easy to see that the conditional distribution of Wp given Zp = z is Gaussian with mean b∞,pz and variance
σ2ζ2∞,p. Consequently, the probability of the event {|Wp + νp| ≥ cpσξ∞,p} conditional on Zp = z is given by
the integrand shown in (15). Since Zp has distribution Φ∞,p as noted above, it follows that (37) and (15)
agree.
Remark A.6 If C
∞ = 0 for p > p∗, then in view of the above discussion Zp∗ = Zp = ZP , and hence
Φ∞,p∗ = Φ∞,p = Φ∞,P , holds for all p > p∗. Using the independence of Wr, r > p∗, from Zp∗ , inspection of
(37) shows that G∞,θ,σ,γ reduces to Φ∞,P ; see also Leeb (2006, Remark 5.2).
Proof of Lemma A.4: From (37) (or (15)) it follows that the map γ 7→ G∞,θ,σ,γ(t) depends only on t,
Q, A, σ, the critical values cp for O < p ≤ P , as well as on θ; however, the dependence on θ is only through
p∗ = max{p0(θ),O}. It hence suffices to find, for each possible value of p∗ in the range p∗ = O, . . . , q∗ − 1,
constants 0 < ρ0 < ∞ and δ0 > 0 such that (35) is satisfied for some (and hence all) θ returning this
particular value of p∗ = max{p0(θ),O}. For this in turn it is sufficient to show that for every θ ∈Mq∗−1 the
quantity G∞,θ,σ,γ(t) is non-constant as a function of γ ∈Mq∗\Mq∗−1.
Let θ ∈Mq∗−1 and assume that G∞,θ,σ,γ(t) is constant in γ ∈Mq∗\Mq∗−1. Observe that, by assumption,
∞ is non-zero while C
∞ = 0 for p > q
∗. For γ ∈ Mq∗ , we clearly have νq∗ = γq∗ and νr = 0 for r > q∗.
Letting γq∗−1 → ∞ while γq∗ is held fixed, we see that νq∗−1 → ∞; hence,
P(|Wq∗−1 + νq∗−1| < cq∗−1σξ∞,q∗−1) → 0.
It follows that (37) converges to
Zq∗−1 ≤ t+ ξ−2∞,q∗C(q
∞ γq∗
P(|Wq∗ + γq∗ | < cq∗σξ∞,q∗)
q=q∗+1
P(|Wq| < cqσξ∞,q)
Zq∗ ≤ t, |Wq∗ + γq∗ | ≥ cq∗σξ∞,q∗
q=q∗+1
P(|Wq| < cqσξ∞,q) (38)
p=q∗+1
Zp ≤ t, |Wp| ≥ cpσξ∞,p
q=p+1
P(|Wq| < cqσξ∞,q).
By assumption, the expression in the above display is constant in γq∗ ∈ R\{0}. Dropping the terms that do
not depend on γq∗ and observing that P(|Wq| < cqσξ∞,q) is never zero for q > q∗ > O, we see that
Zq∗−1 ≤ t+ ξ−2∞,q∗C(q
∞ γq∗
P(|Wq∗ + γq∗ | < cq∗σξ∞,q∗)
Zq∗ ≤ t, |Wq∗ + γq∗ | ≥ cq∗σξ∞,q∗
has to be constant in γq∗ ∈ R\{0}. We now show that the expression in (39) is in fact constant in γq∗ ∈ R:
Observe first that P(|Wq∗ + γq∗ | < cq∗σξ∞,q∗) is positive and continuous in γq∗ ∈ R; also the probability
Zq∗ ≤ t, |Wq∗ + γq∗ | ≥ cq∗σξ∞,q∗
is continuous in γq∗ ∈ R since Wq∗ , being normal with mean zero and
positive variance, is absolutely continuously distributed. Concerning the remaining term in (39), we note
that Zq∗−1 =MV where M = [ξ
∞ , . . . , ξ
∞,q∗−1C
(q∗−1)
∞ ] and V = (W1, . . . ,Wq∗−1)
′. In case no row of
M is identically zero, Lemma A.3 shows that also P
Zq∗−1 ≤ t+ ξ−2∞,q∗C
∞ γq∗
is continuous in γq∗ ∈ R.
Hence, in this case (39) is indeed constant for all γq∗ ∈ R. In case a row of M is identically zero, define
N = ξ
∞,q∗C
∞ and rewrite the probability in question as P
MV ≤ t+Nγq∗
. Note that (M : N) has full
row-rank k, since
(M : N)diag[ξ2∞,1, . . . , ξ
∞,q∗ ](M : N)
ξ−2∞,rC
ξ−2∞,rC
∞ = AQ
−1A′ (40)
by definition of q∗ and since the latter matrix is non-singular in view of rank A = k. Lemma A.2 then shows
that there exists a γ
q∗ ∈ R, z ∈ {−1, 1}, and a constant c > 0 such that P
MV ≤ t+N(γ(0)q∗ − δz)
and P
MV ≤ t+N(γ(0)q∗ + δz)
≥ c holds for arbitrary small δ > 0. Observe that γ(0)q∗ − δz as well as
q∗ − δz are non-zero for sufficiently small δ > 0. But then (39) – being constant for γq∗ ∈ R\{0} – gives
the same value for γq∗ = γ
q∗ − δz and γq∗ = γ
q∗ + δz and all sufficiently small δ > 0. Letting δ go to zero
in this equality and using the continuity properties for the second and third probability in (39) noted above
we obtain that
cP(|Wq∗ + γ(0)q∗ | < cq∗σξ∞,q∗) + P
Zq∗ ≤ t, |Wq∗ + γ(0)q∗ | ≥ cq∗σξ∞,q∗
≤ lim inf
Zq∗−1 ≤ t+ ξ−2∞,q∗C(q
q∗ + δz)
P(|Wq∗ + γ(0)q∗ | < cq∗σξ∞,q∗)
Zq∗ ≤ t, |Wq∗ + γ(0)q∗ | ≥ cq∗σξ∞,q∗
= lim inf
Zq∗−1 ≤ t+ ξ−2∞,q∗C(q
q∗ − δz)
P(|Wq∗ + γ(0)q∗ | < cq∗σξ∞,q∗)
Zq∗ ≤ t, |Wq∗ + γ(0)q∗ | ≥ cq∗σξ∞,q∗
Zq∗ ≤ t, |Wq∗ + γ(0)q∗ | ≥ cq∗σξ∞,q∗
which is impossible since c > 0 and P(|Wq∗ + γ(0)q∗ | < cq∗σξ∞,q∗) > 0. Hence we have shown that (39) is
indeed constant for all γq∗ ∈ R.
Now write Z,W , C, τ , and x for Zq∗−1−t, −Wq∗/σξ∞,q∗ , σξ
∞,q∗C
∞ , cq∗ , and γq∗/σξ∞,q∗ , respectively.
Upon observing that Zq∗ equals Zq∗−1 + ξ
∞,q∗C
∞ Wq∗ , it is easy to see that (39) can be written as in (31).
By our assumptions, this expression is constant in x = γq∗/σξ∞,q∗ ∈ R. Lemma A.1 then entails that either
C = 0 or that P(Z ≤ Cx) = 0 for each x ∈ R. Since C equals σξ−1∞,q∗C
∞ , it is non-zero by assumption.
Hence,
Zq∗−1 ≤ t+ ξ−2∞,q∗C(q
∞ γq∗
must hold for every value of γq∗ . But the above probability is just the conditional probability that Zq∗ ≤ t
given Wq∗ = −γq∗ . It follows that P(Zq∗ ≤ t) equals zero as well. By our assumption C
∞ = 0 for p > q
and hence Zq∗ = ZP . We thus obtain P(ZP ≤ t) = 0, a contradiction with the fact that ZP is a Gaussian
random variable on Rk with non-singular variance-covariance matrix σ2AQ−1A′. ✷
Inspection of the above proof shows that it can be simplified if the claim of non-constancy of G∞,θ,σ,γ(t)
as a function of γ ∈Mq∗\Mq∗−1 in Lemma A.4 is weakened to non-constancy for γ ∈Mq∗ . The strong form
of the lemma as given here is needed in the proof of Proposition B.1.
Proof of Lemma A.5: Let p⊕ be the largest index p, O ≤ p ≤ P , for which A[p] has a row of zeroes, and
set p⊕ = 0 if no such index exists. We first show that p⊕ satisfies p⊕ < q
∗. Suppose p⊕ ≥ q∗ would hold. Since
Zp⊕ is a Gaussian random vector with mean zero and variance-covariance matrix σ
2A[p⊕]Q[p⊕ : p⊕]
−1A[p⊕]
at least one component of Zp⊕ is equal to zero with probability one. However, Zp⊕ equals ZP because of
p⊕ ≥ q∗ and the definition of q∗. This leads to a contradiction since ZP has the non-singular variance-
covariance matrix σ2AQ−1A′. Without loss of generality, we may hence assume that p⊙ = p⊕.
In view of the discussion in the first paragraph of the proof of Lemma A.4, it suffices to establish, for
each possible value s in the range O ≤ s ≤ p⊙, the result (36) for some θ with s = max{p0(θ),O} = p∗. Now
fix such an s and θ (as well as, of course, t, Q, A, σ, and the critical values cp for O < p ≤ P ). Then (37)
expresses the map γ 7→ G∞,θ,σ,γ(t) in terms of ν = (ν1, . . . , νP )′. It is easy to see that the correspondence
between γ and ν is a linear bijection from RP onto itself, and that γ ∈ Mq∗ if and only if ν ∈ Mq∗ . It is
hence sufficient to find a δ∗ > 0 and vectors ν and µ in Mq∗ such that (37) with ν + ǫµ in place of ν and
(37) with ν − ǫµ in place of ν differ by at least δ∗ for sufficiently small ǫ > 0. Note that (37) is the sum of
P − p∗ + 1 terms indexed by p = p∗, . . . , P . We shall now show that ν and µ can be chosen in such a way
that, when replacing ν with ν + ǫµ and ν − ǫµ, respectively, (i) the resulting terms in (37) corresponding to
p = p⊙ differ by some d > 0, while (ii) the difference of the other terms becomes arbitrarily small, provided
that ǫ > 0 is sufficiently small.
Consider first the case where s = p∗ = p⊙. Using the shorthand notation
g(ν) = P
Zp⊙ ≤ t+
r=p⊙+1
ξ−2∞,rC
note that the p⊙-th term in (37) is given by g(ν) multiplied by a product of positive probabilities which are
continuous in ν. To prove property (i) it thus suffices to find a constant c > 0, and vectors ν and µ in Mq∗
such that |g(ν + ǫµ)− g(ν − ǫµ)| ≥ c holds for each sufficiently small ǫ > 0.
In the sub-case p⊙ = 0 choose c = 1, set
ν = −[C(1)∞ , . . . , C(P )∞ ]′
ξ−2∞,rC
µ = [C(1)∞ , . . . , C
ξ−2∞,rC
(1, . . . , 1)′,
observing that the matrix to be inverted is indeed non-singular, since – as discussed after Lemma A.5 – it
is up to a multiplicative factor σ2 identical to the variance-covariance matrix σ2AQ−1A′ of ZP . But then
ν and µ satisfy
r=p⊙+1
ξ−2∞,rC
∞ νr = −t and
r=p⊙+1
ξ−2∞,rC
∞ µr = (1, . . . , 1)
′ if we note that by the
definition of q∗
r=p⊙+1
ξ−2∞,rC
∞ νr =
ξ−2∞,rC
holds and that a similar relation holds with µ replacing ν. Since Zp⊙ = Z0 = 0 ∈ Rk, it is then obvious that
g(ν + ǫµ) and g(ν − ǫµ) differ by 1 for each ǫ > 0.
In the other sub-case p⊙ > 0, define M = [ξ
∞ , . . . , ξ
∞,p⊙C
∞ ], N =
[ξ−2∞,p⊙+1C
(p⊙+1)
∞ , . . . , ξ
∞,q∗C
∞ ], and V = (W1, . . . ,Wp⊙)
′. It is then easy to see that g(ν) equals
f((νp⊙+1, . . . , νq∗)
′), with f defined as in Lemma A.2, and that M has a row of zeros. Furthermore, the
matrix (M : N) has rank k by the same argument as in the proof of Lemma A.4; cf. (40). By Lemma A.2,
we thus obtain vectors x0 and z, and a c > 0 such that |f(x0+ ǫz)−f(x0− ǫz)| ≥ c holds for each sufficiently
small ǫ > 0. Setting (νp⊙+1, . . . , νq∗)
′ = x0, (µp⊙+1, . . . , µq∗)
′ = z, setting ν[¬q∗], and µ[¬q∗] each equal to
zero, and setting ν[p⊙] and µ[p⊙] to arbitrary values, we see that g(ν ± ǫµ) has the desired properties.
To complete the proof in case s = p∗ = p⊙, we need to establish property (ii) for which it suffices to
show that, for p > p⊙, the p-th term in (37) depends continuously on ν. For p > q
∗, the p-th term does not
depend on ν, because C
∞ = 0 for r = q∗, . . . , P . For p satisfying p⊙ < p ≤ q∗, it suffices to show that
h(νp, . . . , νq∗) = P
Zp ≤ t+
r=p+1
∞ νr, |Wp + νp| ≥ cpσξ∞,p
is a continuous function. Suppose that (ν
p , . . . , ν
q∗ ) converges to (νp, . . . , νq∗) as m→ ∞. For arbitrary
α > 0,
r=p+1 ξ
∞ νr and
r=p+1 ξ
r differ by less than α in each coordinate, provided that
m is sufficiently large. This implies
lim sup
h(ν(m)p , . . . , ν
≤ lim sup
P(Zp ≤ t+
r=p+1
∞ νr + α(1, . . . , 1)
′, |Wp + ν(m)p | ≥ cpσξ∞,p)
= P(Zp ≤ t+
r=p+1
ξ−2∞,rC
∞ νr + α(1, . . . , 1)
′, |Wp + νp| ≥ cpσξ∞,p),
observing that the latter probability is obviously continuous in the single variable νp (since Wp has an ab-
solutely continuous distribution). Letting α decrease to zero we obtain lim supm→∞ h(ν
p , . . . , ν
q∗ ) ≤
h(νp, . . . , νq∗). A similar argument establishes lim infm→∞ h(ν
p , . . . , ν
q∗ ) ≥ P(Zp < t +
r=p+1 ξ
∞ νr, |Wp + νp| ≥ cpσξ∞,p). The proof of the continuity of h is then complete if we can show
that P
Zp ≤ ·, |Wp + νp| ≥ cpσξ∞,p
is continuous or, equivalently, that P
Zp ≤ ·
∣∣|Wp + νp| ≥ cpσξ∞,p
is a continuous cdf. Since p > p⊙, the variance-covariance matrix σ
2A[p]Q[p : p]−1A[p]′ of Zp does only
have non-zero diagonal elements. Consequently, when representing Zp as B(W1, . . . ,Wp)
′, the matrix B
cannot have rows that consist entirely of zeros. The conditional distribution of (W1, . . . ,Wp)
′ given the
event {|Wp + νp| ≥ cpσξ∞,p} is clearly absolutely continuous w.r.t. p-dimensional Lebesgue measure. But
then Lemma A.3 delivers the desired result.
The case where s = p∗ < p⊙ is reduced to the previously discussed case as follows: It is easy to see that,
for νp⊙ → ∞, the expression in (37) converges to a limit uniformly w.r.t. all νp with p 6= p⊙. Then observe
that this limit is again of the form (37) but now with p⊙ taking the rôle of p∗. ✷
B Non-Uniformity of the Convergence of the Finite-Sample Cdf
to the Large-Sample Limit
Proposition B.1 a. Suppose that Aθ̃(q) and θ̃q(q) are asymptotically correlated, i.e., C
∞ 6= 0, for some
q satisfying O < q ≤ P , and let q∗ denote the largest q with this property. Then for every θ ∈Mq∗−1,
every σ, 0 < σ <∞, and every t ∈ Rk there exists a ρ, 0 < ρ <∞, such that
lim inf
ϑ∈Mq∗
||ϑ−θ||<ρ/
|Gn,ϑ,σ(t)−G∞,ϑ,σ(t)| > 0 (41)
holds. The constant ρ may be chosen in such a way that it depends only on t, Q, A, σ, and the critical
values cp for O < p ≤ P .
b. Suppose that Aθ̃(q) and θ̃q(q) are asymptotically uncorrelated, i.e., C
∞ = 0, for all q satisfying O <
q ≤ P . Then Gn,θ,σ converges to Φ∞,P in total variation uniformly in θ ∈ RP ; more precisely
σ∗≤σ≤σ∗
||Gn,θ,σ − Φ∞,P ||TV
n→∞−→ 0
holds for any constants σ∗ and σ
∗ satisfying 0 < σ∗ ≤ σ∗ <∞.
Under the assumptions of Proposition B.1(a), we see that convergence of Gn,θ,σ(t) to G∞,θ,σ(t) is non-
uniform over shrinking ‘tubes’ aroundMq∗−1 that are contained in Mq∗ . [On the complement of a tube with
a fixed positive radius, i.e., on the set U = {θ ∈ RP : |θq∗ | ≥ r} with fixed r > 0, convergence of Gn,θ,σ(t)
to G∞,θ,σ(t) is in fact uniform (even with respect to the total variation distance), as can be shown. Note
that for θ ∈ U the cdf G∞,θ,σ(t) reduces to the Gaussian cdf Φ∞,P (t), i.e., to the asymptotic distribution of
the least-squares estimator based on the overall model; cf. Remark A.6.] A precursor to Proposition B.1(a)
is Corollary 5.5 of Leeb and Pötscher (2003) which establishes (41) in the special case where O = 0 and
where A is the P × P identity matrix. Proposition B.1(b) describes an exceptional case where convergence
is uniform. [In this case G∞,θ,σ reduces to the Gaussian cdf Φ∞,P for all θ and Φ∞,P = Φ∞,p, O ≤ p ≤ P ,
holds; cf. Remark A.6.] Recall that under the assumptions of part (b) of Proposition B.1 we necessarily
always have (i) O > 0, and (ii) rank A[O] = k; cf. Proposition 4.4 in Leeb and Pötscher (2006b).
Proof of Proposition B.1: We first prove part (a). As noted at the beginning of the proof of Lemma
A.4, the map γ 7→ G∞,θ,σ,γ(t) depends only on t, Q, A, σ, the critical values cp for O < p ≤ P , as well as on
θ, but the dependence on θ is only through p∗ = max{p0(θ),O}. It hence suffices to find, for each possible
value of p∗ in the range p∗ = O, . . . , q∗ − 1, a constant 0 < ρ < ∞ such that (41) is satisfied for some (and
hence all) θ returning this particular value of p∗ = max{p0(θ),O}. For this in turn it is sufficient to show
that given such a θ we can find a γ ∈Mq∗ such that
lim inf
|Gn,θ+γ/√n,σ(t)−G∞,θ+γ/√n,σ(t)| > 0 (42)
holds. Note that (42) is equivalent to
lim inf
|G∞,θ,σ,γ(t)−G∞,θ+γ/√n,σ(t)| > 0 (43)
in light of Proposition 2.1. To establish (43), we proceed as follows: For each γ ∈ Mq∗ with γq∗ 6= 0,
G∞,θ+γ/
n,σ(t) in (15) reduces to Φ∞,q∗(t) as is easily seen from (37) since p0(θ+γ/
n) = q∗ which in turn
follows from p0(θ) < q
∗ and γq∗ 6= 0. Furthermore, Lemma A.4 entails that G∞,θ,σ,γ(t) is non-constant in
γ ∈Mq∗\Mq∗−1. But this shows that (43) must hold.
To prove part (b), we write
||Gn,θ,σ − Φ∞,P ||TV =
∣∣∣∣∣∣
∣∣∣∣∣∣
Gn,θ,σ(·|p)πn,θ,σ(p)− Φ∞,P (·)
∣∣∣∣∣∣
∣∣∣∣∣∣
||Gn,θ,σ(·|p)− Φ∞,P (·)||TV πn,θ,σ(p),
where the conditional cdfs Gn,θ,σ(·|p) and the model selection probabilities πn,θ,σ(p) have been introduced
after (12). By the ‘uncorrelatedness’ assumption, we have that Φ∞,p = Φ∞,P for all p in the rangeO ≤ p ≤ P ;
cf. Remark A.6. We hence obtain
σ∗≤σ≤σ∗
||Gn,θ,σ − Φ∞,P ||TV ≤
σ∗≤σ≤σ∗
||Gn,θ,σ(·|p)− Φ∞,p(·)||TV πn,θ,σ(p). (44)
Now for every p with O ≤ p ≤ P and for every ρ, 0 < ρ <∞, we can write
σ∗≤σ≤σ∗
||Gn,θ,σ(·|p)− Φ∞,p(·)||TV πn,θ,σ(p)
≤ max
‖θ[¬p]‖<ρ/
σ∗≤σ≤σ∗
||Gn,θ,σ(·|p)− Φ∞,p(·)||TV , sup
‖θ[¬p]‖≥ρ/
σ∗≤σ≤σ∗
πn,θ,σ(p)
. (45)
In case p = P , we use here the convention that the second term in the maximum is absent and that the first
supremum in the first term in the maximum extends over all of RP . Letting first n and then ρ go to infinity
in (45), we may apply Lemmas C.2 and C.3 in Leeb and Pötscher (2005b) to conclude that the l.h.s. of (45),
and hence the l.h.s. of (44), goes to zero as n→ ∞. ✷
C Proofs for Sections 2.1 to 2.2.2
In the proofs below it will be convenient to show the dependence of Φn,p and Φ∞,p on σ in the notation. Thus,
in the following we shall write Φn,p,σ and Φ∞,p,σ, respectively, for the cdf of a k-variate Gaussian random
vector with mean zero and variance-covariancematrix σ2A[p](X [p]′X [p]/n)−1A[p]′ and σ2A[p]Q[p : p]−1A[p]′,
respectively. For convenience, let Φn,0,σ and Φ∞,0,σ denote the cdf of point-mass at zero in R
The following lemma is elementary to prove, if we recall that bn,pz converges to b∞,pz as n → ∞ for
every z ∈ ImA[p], the column space of A[p].
Lemma C.1 Suppose p > O. Define Rn,p(z, σ) = 1 − ∆σζn,p(bn,pz, cpσξn,p) and R∞,p(z, σ) = 1 −
∆σζ∞,p(b∞,pz, cpσξ∞,p) for z ∈ ImA[p], 0 < σ <∞. Let σ
(n) converge to σ, 0 < σ < ∞. If ζ∞,p 6= 0, then
Rn,p(z, σ
(n)) converges to R∞,p(z, σ) for every z ∈ ImA[p]; if ζ∞,p = 0, then convergence holds for every
z ∈ ImA[p], except possibly for z ∈ ImA[p] satisfying |b∞,pz| = cpσξ∞,p. [This exceptional subset of ImA[p]
has rank(A[p])-dimensional Lebesgue measure zero since cpσξ∞,p > 0.]
The following observation is useful in the proof of Proposition 2.2 below: Since the proposition depends
on Y only through its distribution (cf. Remark 4.1), we may assume without loss of generality that the
errors in (5) are given by ut = σεt, t ∈ N, with i.i.d. εt that are standard normal. In particular, all random
variables involved are then defined on the same probability space.
Proof of Proposition 2.2: Since Pn,θ,σ(p̄ = p0(θ)) → 1 by consistency, we may replace max{p̄,O}
by p∗ = max{p0(θ),O} in the formula for Ǧn for the remainder of the proof. Furthermore, since σ̂ → σ
in Pn,θ,σ-probability, each subsequence contains a further subsequence along which σ̂ → σ almost surely
(with respect to the probability measure on the common probability space supporting all random variables
involved), and we restrict ourselves to such a further subsequence for the moment. In particular, we write
{σ̂ → σ} for the event that σ̂ converges to σ along the subsequence under consideration; clearly, the event
{σ̂ → σ} has probability one. Also note that we can assume without loss of generality that σ̂ > 0 holds
on this event (at least from some data-dependent n onwards), since σ > 0 holds. But then obviously
q=p∗+1
∆σ̂ξn,q (0, cqσ̂ξn,q) converges to
q=p∗+1
∆σξ∞,q (0, cqσξ∞,q), and Φ̂n,p∗(t) converges to Φ∞,p∗,σ(t)
in total variation by Lemma A.3 of Leeb (2005) in case p∗ > 0, and trivially so in case p∗ = 0. This proves
that the first term in the formula for Ǧn converges to the corresponding term in the formula for G∞,θ,σ in
total variation.
Next, consider the term in Ǧn that carries the index p > p∗. By Lemma A.3 in Leeb (2005), Φ̂n,p = Φn,p,σ̂
has a density dΦn,p,σ̂/dΦ∞,p,σ with respect to Φ∞,p,σ, which converges to 1 except on a set that has measure
zero under Φ∞,p,σ. By Scheffé’s Lemma (Billingsley (1995), Theorem 16.12), dΦn,p,σ̂/dΦ∞,p,σ converges to 1
also in the L1(Φ∞,p,σ)-sense. By Lemma C.1, Rn,p(z, σ̂) converges to R∞,p(z, σ) except possibly on a set that
has measure zero under Φ∞,p,σ. (Recall that Φ∞,p,σ is concentrated on ImA[p] and is not degenerate there.)
Observing that |Rn,p(z, σ̂)| is uniformly bounded by 1, we obtain that Rn,p(z, σ̂) converges to R∞,p(z, σ)
also in the L1(Φ∞,p,σ)-sense. Hence,
∥∥∥∥Rn,p(z, σ̂)
dΦn,p,σ̂
dΦ∞,p,σ
(z)−R∞,p(z, σ)
∥∥∥∥Rn,p(z, σ̂)
dΦn,p,σ̂
dΦ∞,p,σ
(z)−Rn,p(z, σ̂)
∥∥∥∥+ ‖Rn,p(z, σ̂)−R∞,p(z, σ)‖ (46)
dΦn,p,σ̂
dΦ∞,p,σ
(z)− 1
∥∥∥∥+ ‖Rn,p(z, σ̂)−R∞,p(z, σ)‖
n→∞−→ 0
where ‖·‖ denotes the L1(Φ∞,p,σ)-norm. Since
q=p+1 ∆σ̂ξn,q (0, cqσ̂ξn,q) obviously converges to∏P
q=p+1 ∆σξ∞,q (0, cqσξ∞,q), the relation (46) shows that the term in Ǧn carrying the index p converges
to the corresponding term in G∞,θ,σ in the total variation sense. This proves (18) along the subsequence
under consideration. However, since any subsequence contains such a further subsequence, this establishes
(18). Since Gn,θ,σ converges to G∞,θ,σ in total variation by Proposition 2.1, the claim in (17) also follows.✷
Before we prove the main result we observe that the total variation distance between Pn,θ,σ and
Pn,ϑ,σ satisfies ||Pn,θ,σ − Pn,ϑ,σ||TV ≤ 2Φ(‖θ − ϑ‖λ
max(X
′X)/2σ) − 1; furthermore, if θ(n) and ϑ(n) sat-
∥∥∥θ(n) − ϑ(n)
∥∥∥ = O(n−1/2), the sequence Pn,ϑ(n),σ is contiguous with respect to the sequence Pn,θ(n),σ
(and vice versa). This follows exactly in the same way as Lemma A.1 in Leeb and Pötscher (2006a).
Proof of Theorem 2.3: We first prove (20) and (21). For this purpose we make use of Lemma 3.1 in
Leeb and Pötscher (2006a) with α = θ ∈ Mq∗−1, B = Mq∗ , Bn = {ϑ ∈ Mq∗ : ‖ϑ− θ‖ < ρ0n−1/2}, β = ϑ,
ϕn(β) = Gn,ϑ,σ(t), ϕ̂n = Ĝn(t), where ρ0, 0 < ρ0 < ∞, will be chosen shortly (and σ is held fixed). The
contiguity assumption of this lemma (as well as the mutual contiguity assumption used in the corrigendum
to Leeb and Pötscher (2006a)) is satisfied in view of the preparatory remark above. It hence remains only to
show that there exists a value of ρ0, 0 < ρ0 < ∞, such that δ
in Lemma 3.1 of Leeb and Pötscher (2006a)
(which represents the limit inferior of the oscillation of ϕn(·) over Bn) is positive. Applying Lemma 3.5(i)
of Leeb and Pötscher (2006a) with ζn = ρ0n
−1/2 and the set G0 equal to the set G, it remains, in light
of Proposition 2.1, to show that there exists a ρ0, 0 < ρ0 < ∞, such that G∞,θ,σ,γ(t) as a function of γ is
non-constant on the set {γ ∈ Mq∗ : ‖γ‖ < ρ0}. In view of Lemma 3.1 of Leeb and Pötscher (2006a), the
corresponding δ0 can then be chosen as any positive number less than one-half of the oscillation of G∞,θ,σ,γ(t)
over this set. That such a ρ0 indeed exists follows now from Lemma A.4 in Appendix A, where it is also
shown that ρ0 and δ0 can be chosen such that they depend only on t, Q,A, σ, and cp for O < p ≤ P . This
completes the proof of (20) and (21).
To prove (22) we use Corollary 3.4 in Leeb and Pötscher (2006a) with the same identification of
notation as above, with ζn = ρ0n
−1/2, and with V = Mq∗ (viewed as a vector space isomorphic to
). The asymptotic uniform equicontinuity condition in that corollary is then satisfied in view of
||Pn,θ,σ − Pn,ϑ,σ||TV ≤ 2Φ(‖θ − ϑ‖λ
max(X
′X)/2σ) − 1. Given that the positivity of δ∗ has already be es-
tablished in the previous paragraph, applying Corollary 3.4(i) in Leeb and Pötscher (2006a) then establishes
(22). ✷
Proof of Remark 2.4: The proof is similar to the proof of (22) just given, except for using Corol-
lary 3.4(ii) and Lemma 3.5(ii) in Leeb and Pötscher (2006a) instead of Corollary 3.4(i) and Lemma 3.5(i)
from that paper. Furthermore, Lemma A.5 in Appendix A instead of Lemma A.4 is used. ✷
Proof of Proposition 2.5: In view of Proposition B.1(b) and the fact that Φ̂n,P (·) = Φn,P,σ̂(·) holds
(in case σ̂ > 0), it suffices to show that
σ∗≤σ≤σ∗
||Φn,P,σ(·)− Φ∞,P,σ(·)||TV
n→∞−→ 0 (47)
σ∗≤σ≤σ∗
Pn,θ,σ
||Φn,P,σ̂(·)− Φn,P,σ(·)||TV > δ
) n→∞−→ 0 (48)
hold for each δ > 0, and for any constants σ∗ and σ
∗ satisfying 0 < σ∗ ≤ σ∗ <∞. [Note that the probability
in (48) does in fact not depend on θ.] But this has already been established in the proof of Proposition 4.3
of Leeb and Pötscher (2005b). ✷
D Proofs for Section 3
Proof of Theorem 3.1: After rearranging the elements of θ (and hence the regressors) if necessary and
then correspondingly rearranging the rows of the matrix A, we may assume without loss of generality that
r∗ = (1, . . . , 1, 0), and hence that i(r∗) = P . That is, Mr∗ = MP−1 and Mrfull = MP . Furthermore, note
that after this arrangement C
∞ 6= 0. Let p̂ be the model selection procedure introduced in Section 2 with
O = P − 1, cP = c, and cO = 0. Let θ̃ be the corresponding post-model-selection estimator and let Gn,θ,σ(t)
be as defined in Section 2.1. Condition (24) now implies: For every θ ∈ MP−1 which has exactly P − 1
non-zero coordinates
Pn,θ,σ ({r̂ = rfull}N{p̂ = P}) = lim
Pn,θ,σ ({r̂ = r∗}N{p̂ = P − 1}) = 0 (49)
holds for every 0 < σ < ∞. Since the sequences Pn,ϑ(n),σ and Pn,θ,σ are contiguous for ϑ
(n) satisfying∥∥∥θ − ϑ(n)
∥∥∥ = O(n−1/2) as remarked prior to the proof of Theorem 2.3 in Appendix C, it follows that
condition (49) continues to hold with Pn,ϑ(n),σ replacing Pn,θ,σ. This implies that for every sequence of
positive real numbers sn with sn = O(n
−1/2), for every σ, 0 < σ < ∞, and for every θ ∈ MP−1 which has
exactly P − 1 non-zero coordinates
||ϑ−θ||<sn
‖Kn,ϑ,σ −Gn,ϑ,σ‖TV → 0 (50)
holds as n → ∞. From (50) we conclude that the limit of Kn,θ+γ/√n,σ (with respect to total variation
distance) exists and coincides with G∞,θ,σ,γ . Repeating the proof of Theorem 2.3 with q
∗ = P , with
Kn,ϑ,σ(t) replacing Gn,ϑ,σ(t), and with K̂n(t) replacing Ĝn(t) gives the desired result. ✷
E References
Ahmed, S. E. & A. K. Basu (2000): Least squares, preliminary test and Stein-type estimation in general
vector AR(p) models. Statistica Neerlandica 54, 47–66.
Bauer, P., Pötscher, B. M. & P. Hackl (1988): Model selection by multiple test procedures. Statistics
19, 39–44.
Billingsley, P. (1995): Probability and Measure, (3rd ed.). Wiley.
Brownstone, D. (1990): Bootstrapping improved estimators for linear regression models. Journal of
Econometrics 44, 171–187.
Danilov, D. L. & J. R. Magnus (2004): On the harm that ignoring pre-testing can cause. Journal of
Econometrics 122, 27–46.
Dijkstra, T. K. & J. H. Veldkamp (1988): ‘Data-driven selection of regressors and the bootstrap’. Lecture
Notes in Economics and Mathematical Systems 307, 17–38.
Dukić, V. M. & E. A Peña (2002): Variance estimation in a model with gaussian submodel. Journal of
the American Statistical Association 100, 296-309.
Freedman, D. A., Navidi, W. & S. C. Peters (1988): ‘On the impact of variable selection in fitting
regression equations’. Lecture Notes in Economics and Mathematical Systems 307, 1–16.
Hansen, P. R. (2003): Regression analysis with many specifications: a bootstrap method for robust
inference. Working Paper, Department of Economics, Brown University.
Hjort, N. L. & G. Claeskens (2003): Frequentist model average estimators. Journal of the American
Statistical Association 98, 879–899.
Kabaila, P. (1995): The effect of model selection on confidence regions and prediction regions. Econo-
metric Theory 11, 537–549.
Kapetanios, G. (2001): Incorporating lag order selection uncertainty in parameter inference for AR
models. Economics Letters 72, 137–144.
Kilian, L. (1998): Accounting for lag order uncertainty in autoregressions: the endogenous lag order
bootstrap algorithm. Journal of Time Series Analysis 19, 531–548.
Knight, K. (1999): Epi-convergence in distribution and stochastic equi-semicontinuity. Working Paper,
Department of Statistics, University of Toronto.
Kulperger, R. J. & S. E. Ahmed (1992): A bootstrap theorem for a preliminary test estimator. Commu-
nications in Statistics: Theory and Methods 21, 2071–2082.
Leeb, H. (2002): On a differential equation with advanced and retarded arguments. Communications on
Applied Nonlinear Analysis 9, 77–86.
Leeb, H. (2005): The distribution of a linear predictor after model selection: conditional finite-sample
distributions and asymptotic approximations. Journal of Statistical Planning and Inference 134, 64–89.
Leeb, H. (2006): The distribution of a linear predictor after model selection: unconditional finite-sample
distributions and asymptotic approximations. IMS Lecture Notes-Monograph Series 49, 291–311.
Leeb, H. & B. M. Pötscher (2003): The finite-sample distribution of post-model-selection estimators and
uniform versus nonuniform approximations. Econometric Theory 19, 100–142.
Leeb, H. & B. M. Pötscher (2005a): Model selection and inference: facts and fiction. Econometric Theory
21, 21–59.
Leeb, H. & B. M. Pötscher (2005b): Can one estimate the conditional distribution of post-model-selection
estimators? Working Paper, Department of Statistics, University of Vienna.
Leeb, H. & B. M. Pötscher (2006a): Performance limits for estimators of the risk or distribution of
shrinkage-type estimators, and some general lower risk bound results. Econometric Theory 22, 69-97.
(Corrigendum. Econometric Theory, forthcoming.)
Leeb, H. & B. M. Pötscher (2006b): Can one estimate the conditional distribution of post-model-selection
estimators? Annals of Statistics 34, 2554-2591.
Lehmann, E. L. & G. Casella (1998): Theory of Point Estimation, 2nd Edition, Springer Texts in
Statistics. Springer-Verlag.
Nickl, R. (2003): Asymptotic Distribution Theory of Post-Model-Selection Maximum Likelihood Estima-
tors. Masters Thesis, Department of Statistics, University of Vienna.
Pötscher, B. M. (1991): Effects of model selection on inference. Econometric Theory 7, 163–185.
Pötscher, B. M. (1995): Comment on ‘The effect of model selection on confidence regions and prediction
regions’ by P. Kabaila. Econometric Theory 11, 550–559.
Pötscher, B. M. & A. J. Novak (1998): The distribution of estimators after model selection: large and
small sample results. Journal of Statistical Computation and Simulation 60, 19–56.
Rao, C. R. & Y. Wu (2001): ‘On model selection,’ IMS Lecture Notes-Monograph Series 38, 1–57.
Robinson, G. K. (1979): Conditional properties of statistical procedures, Annals of Statistics 7, 742–755.
Sen, P. K. (1979): Asymptotic properties of maximum likelihood estimators based on conditional speci-
fication. Annals of Statistics 7, 1019–1033.
Sen P. K. & A. K. M. E. Saleh (1987): On preliminary test and shrinkage M-estimation in linear models.
Annals of Statistics 15, 1580–1592.
van der Vaart, A. W. (1998): Asymptotic Statistics. Cambridge University Press.
