Rejoinder on: Minimal penalties and the slope heuristics: a survey <s> R.4.2. Time-series analysis <s> In this paper, we address the problem of model selection by minimization of a penalized criterion, for non necessary independent random variables. The penalty is obtained by a bloc-resampling estimator of the ideal penalty. We also propose to optimize the leading constant in this penalty, using the slope algorithm. When the data are $\beta$ or $\tau$-mixing, our estimators satisfy oracle inequalities with leading constant asymptotically equal to $1$ and we prove that the slope heuristic holds. <s> BIB001 </s> Rejoinder on: Minimal penalties and the slope heuristics: a survey <s> R.4.2. Time-series analysis <s> We introduce a general approach to prove oracle properties in context tree selection. The results derive from a concentration condition that is verified, for example, by mixing processes. Moreover, we show the superiority of the oracle approach from a non-asymptotic point of view in simulations where the classical BIC estimator has nice oracle properties even when it does not recover the source. ::: Our second objective is to extend the slope algorithm of \cite{AM08} to context tree estimation. The algorithm gives a practical way to evaluate the leading constant in front of the penalties. We study the slope heuristic underlying this algorithm and obtain the first results on the slope phenomenon in a discrete, non i.i.d framework. We illustrate in simulations the improvement of the oracle properties of BIC estimators by the slope algorithm. <s> BIB002
I would like to thank Adrien Saumard for pointing out an important open problem, which is to develop minimal-penalty algorithms for time-series analysis, for which cross-validation and resampling procedures can be difficult to use for estimator selection. A subsection on this topic was indeed missing in Section 8.3. In addition to the references provided by Adrien Saumard and a few results mentioned in the survey that deal with dependent data BIB001 BIB002 , a framework in which theoretical results can be obtained with dependent data is the one of Section 2 with assumption (HG) replaced by ε ∼ N (0, Σ) for some general covariance matrix Σ. We can always write Σ = A ⊤ A with A ∈ M n (R) symmetric, so that ε = Aξ with ξ ∼ N (0, I n ). Then, by Eq. (5)-(6), we obtain the expectations of the risk and empirical risk: holds true whatever the noise dependence structure! Furthermore, the proof of Theorem 1 can certainly be extended to this setting, with n , D m replaced by C m = tr(ΣΠ m ), a phase transition at C = 1, and an optimal oracle inequality for C = 2. The key argument is to get an equivalent of Eq. (23)-(24) in the dependent case, which can follow from standard Gaussian concentration results (Arlot and Bach, 2011, Propositions 4 and 6) under appropriate assumptions on Σ and (Π m ) m∈M . We leave this problem as an exercise to the reader. It seems reasonable to conjecture that the results of Arlot and Bach (2011) on linear estimators can also be extended to dependent data, under appropriate conditions on Σ and the matrices (A m ) m∈M . However, the above result provides theoretical grounds to a slope-heuristics algorithm that can be used only if tr(ΣΠ m ) is known (up to a multiplicative constant) for all m ∈ M . Such an assumption is strong, but it might be satisfied in some specific cases, either exactly -for instance for linear inverse problems with independent homoscedastic noise-or approximately -with assumptions on the dependence structure. Let us finally remark that the above arguments can also serve as a starting point for analyzing the heteroscedatic-regression setting mentioned in point 1 of the first paragraph of Section 8.3.7. Minimal penalties for clustering is an active field of research -for good reasons since crossvalidation cannot directly be applied for clustering, as noticed by Adrien Saumard-, and the discussion reflects it. Christine Keribin provides a new example where the slope heuristics empirically works well for model-based clustering, despite the fact that all models are biased. I do not dare suggest as an additional open problem the validation of the slope heuristics in this setting -with a significant bias for all models-, given the lack of theory available for slope heuristics and model-based clustering. Two discussants show the difficulty of developing a well-founded and practical minimalpenalty algorithm for model-based clustering. Adrien Saumard points out some mathematical challenges that may explain the lack of theory in this framework. I hope his suggests will help solving the problem. Jean-Patrick Baudry reports that some extra-parameters of the algorithms used in practice for model-based clustering may induce some troubles for the slope heuristics, leading to underfitting. Jean-Patrick Baudry's discussion underlines a more general issue: when the (theoretical) estimators s m (D n ) cannot be computed exactly in practice, one should not study the estimatorselection problem among ( s m ) m∈M but the (more difficult) problem of choosing among the practical approximations to these estimators.