0.85pt


Supplementary Appendix
Discussions and Lemmas for the bandwidth selection
Under Assumptions REF  - REF , if \(\sqrt{\frac{\log (n\vee d)}{n\delta }} \vee M_n\eta _1(n) = o(1)\) , and \(M_n^2\sqrt{\log (n\vee d)/n^{\epsilon _1}} = (1)\) , it holds that
\(\begin{aligned}[b]|\widehat{V}(\delta ) - V(\delta )| \lesssim \frac{\psi _1(n,\delta )}{\delta },\end{aligned}\) 
uniformly over all \(\delta \in \Delta \) , where
\(\begin{aligned}[b]\psi _1(n,\delta ) = \Vert ^*\Vert _1^2\bigg (\eta _2(n) \vee \sqrt{\frac{\log (n\vee d)}{n\delta }} \vee M_n\eta _1(n) \bigg ).\end{aligned}\) 


See Section REF  for a detailed proof.
Suppose Assumptions REF -  hold. Choose \(J\)  as a proper kernel function of order \(r\)  satisfying the conditions of \(K\)  in Assumption REF . In addition, assume \(J\)  is \(\ell \)  times continuously differentiable and \(J^{(i)}\)  degenerates at the boundary for \(i = 1,\cdots ,\ell -1\) . If \(M_n\le C\sqrt{nb/\log (nd)}\)  for some constant \(C\)  and \(\frac{\log (n\vee d)}{nb^{2\ell +1}}=o(1)\) , then it holds that
\(\begin{aligned}[b]|\widehat{SB}(\delta ) - SB(\delta )|\lesssim \delta ^{2\ell } \psi _2(n,\delta )\end{aligned}\) 
uniformly over all \(\delta \in \Delta \) , where
\(\begin{aligned}[b]\psi _2(n,\delta ) = \Vert ^*\Vert _1^2\bigg (\sqrt{\frac{\log (n\vee d)}{nb^{2\ell +1}}} \vee (\delta \vee b)^r \vee M_n \eta _1(n)(1 \vee \frac{M_n\eta _1(n)}{\delta ^{\ell }}) \vee \eta _2(n)\bigg ).\end{aligned}\) 


See Section REF  for a detailed proof.
In view of (REF ) and (REF ), the error terms \(\sqrt{\frac{\log (n\vee d)}{nb^{2\ell +1}}}\)  and \((\delta \vee b)^r\)  correspond to the variance and bias when estimating \(A(^*,\delta )\)  in (REF ) with the extra smoothing step. This rate is similar to the case of estimating the \(\ell \) th derivative of a density function using a kernel of order \(r\)  in the context of kernel density estimation under the smoothness Assumption . The terms \(M_n \eta _1(n)(1 \vee \frac{M_n\eta _1(n)}{\delta ^{\ell }})\)  and \(\eta _2(n)\)  come from the plug-in error of \(\widehat{}\)  and \(\widehat{}\) , respectively.

Proofs of Main Results
Proof of Lemma  REF
Recall that
\(\begin{aligned}[b]\nabla \bar{R}_{\delta }^i(^{*}) = w(y_i)\frac{y_i_i}{\delta }K\bigg (\frac{y_i(x_i - ^{*T}_i)}{\delta }\bigg ).\end{aligned}\) 
Let \(T_i =\frac{ \sqrt{\delta }^{*T}(\nabla \bar{R}_\delta ^i(^*) - \nabla R_\delta (^*))}{\sqrt{^{*T}^*^*}}\) , we know by definition \(T_i = 0\) .
Consider
\(\begin{aligned}[b]Var[^{*T}\nabla \bar{R}_\delta ^i(^*)] =&[(^{*T}\nabla \bar{R}_\delta ^i(^*))^2] - (^{*T}\nabla R_\delta (^*))^2.\end{aligned}\) 
Here
\(\begin{aligned}[b][(^{*T}\nabla \bar{R}_\delta ^i(^*))^2] =& \sum _{y \in \lbrace -1,1\rbrace }w(y)^2\int \frac{(^{*T})^2}{\delta ^2}\int K^2(\frac{x - ^{*T}}{\delta })f(x|y,) dxf(y,)d\\=&\sum _{y \in \lbrace -1,1\rbrace }w(y)^2\int \frac{(^{*T})^2}{\delta }\int K^2(u)f(u\delta + ^{*T}|y,) duf(y,)d\\=&\frac{1}{\delta }\sum _{y \in \lbrace -1,1\rbrace }w(y)^2\int (^{*T})^2\int K^2(u)(f(^{*T}|y,) + u\delta f^{\prime }(^{*T}|y,) + o(\delta ))duf(y,)d\\=&\frac{1}{\delta }(^{*T}^*^*(1+ o(1))),\end{aligned}\) 
where the second equality is due to a change of variable, and the third equality is due to Assumption REF . Meanwhile, we know
\(\begin{aligned}[b]^{*T}\nabla R_\delta (^*) =& \sum _{y \in \lbrace -1,1\rbrace }w(y)y\int \frac{(^{*T})}{\delta }\int K(\frac{x - ^{*T}}{\delta })f(x|y,) dxf(y,)d\\=&\sum _{y \in \lbrace -1,1\rbrace }w(y)y\int (^{*T})\int K(u)f(u\delta + ^{*T}|y,) duf(y,)d\\=&\sum _{y \in \lbrace -1,1\rbrace }w(y)y\int (^{*T})\int K(u) \frac{(u\delta )^{\ell }}{\ell !}\bigg (f^{(\ell )}(^{*T}|y,) + ((u\delta )^{\zeta })\bigg )duf(y,)d\\=&\delta ^{\ell }^{*T}^*(1 + o(1)).\end{aligned}\) 
This together with Assumption REF  implies that \(Var[^{*T}\nabla R_\delta ^i(^*)] = \frac{1}{\delta }^{*T}^*^*(1+ o(1))\)  and therefore \(Var(T_i) = 1 + o(1).\)  Now we verify the Lyapunov condition
\(\begin{aligned}[b]\frac{1}{n^{3/2}}\sum _i^n |T_i|^3 =& \frac{1}{n^{3/2}}\sum _i^n\bigg |\frac{ \sqrt{\delta }^{*T}(\nabla \bar{R}_\delta ^i(^*) - \nabla R_\delta (^*))}{\sqrt{^{*T}^*^*}}\bigg |^3.\end{aligned}\) 
By Assumption REF ,
\(\begin{aligned}[b]\bigg |\frac{ \sqrt{\delta }^{*T}(\nabla \bar{R}_\delta ^i(^*) - \nabla R_\delta (^*))}{\sqrt{^{*T}^*^*}}\bigg |^3\lesssim & \delta ^{3/2}\bigg |^{*T}(\nabla \bar{R}_\delta ^i(^*) - \nabla R_\delta (^*))\bigg |^3\\\lesssim &\delta ^{3/2}(|^{*T}\nabla \bar{R}_\delta ^i(^*)|^3 + |^{*T}\nabla R_\delta (^*)|^3),\end{aligned}\) 
where we can show that
\(\begin{aligned}[b]|^{*T}\nabla \bar{R}_\delta ^i(^*)|^3 \le & \Vert ^*\Vert _1^3\sum _{y \in \lbrace -1,1\rbrace }\int \delta \max _{j}\bigg |\frac{y_j}{\delta }K(u)\bigg |^3f(u\delta + ^{*T},y,)dud\\\lesssim &\Vert ^*\Vert _1^3M_n^3/\delta ^2,\end{aligned}\) 
and from Lemma  it's easy to see that the first term on the RHS of (REF ) is dominant. This implies that
\(\begin{aligned}[b]\frac{1}{n^{3/2}}\sum _i^n |T_i|^3 = ((\Vert ^*\Vert _1M_n)^3/(n\delta )^{1/2}).\end{aligned}\) 
Therefore, under the condition of this Lemma the Lyapunov condition holds, which completes the proof by applying Lindeberg Feller Central Limit Theorem.

Proof of Theorem  REF
It suffices to show that \((n\delta )^{1/2}|\widehat{S}_\delta (\widehat{}_0) - S_\delta (^*)| = o_{}(1)\)  where \(\widehat{}_0 = (0,\widehat{\gamma })\) . Here we only show that \( (n\delta )^{1/2}|\widehat{S}_\delta ^{(1)}(\widehat{}_0^{(2)}) - S_\delta (^*)| = o_{}(1)\)  and the desired result shall follow naturally from Lemma REF .
By definition, we have
\(\begin{aligned}[b]&(n\delta )^{1/2}|\widehat{S}_\delta ^{(1)}(\widehat{}_0^{(2)})- S_\delta (^*)|\\=& (n\delta )^{1/2}|\widehat{}^{(1)T}\nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - ^{*T}\nabla R_\delta ^{n_{(1)}}(^*)|\\\le &(n\delta )^{1/2}|^{*T}(\nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - \nabla R_\delta ^{n_{(1)}}(^*))| + (n\delta )^{1/2}|(\widehat{}^{(1)} - ^{*})\nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)})|\\:=&I_1 + I_2.\end{aligned}\) 
The fact that \(^{*T}\nabla ^2_{\cdot }R(^*) = 0\)  and Lemma  implies that
\(\begin{aligned}[b]I_1 =& (n\delta )^{1/2}|^{*T}\nabla ^2_{\cdot }R^{n_{(1)}}_{\delta }(^*)(\widehat{}^{(2)} - ^*)| + o_{}(1)\\\le &(n\delta )^{1/2}\Vert ^*\Vert _1\big (\Vert \nabla ^2_{\cdot }R^{n_{(1)}}_{\delta }(^*) - \nabla ^2_{\cdot }R_{\delta }(^*)\Vert _{\max } + \Vert \nabla ^2_{\cdot }R_{\delta }(^*) - \nabla ^2_{\cdot }R(^*)\Vert _{\max }\big )\Vert \widehat{}^{(2)} - ^*\Vert _1+ o_{}(1)\\\lesssim & (n\delta )^{1/2}\Vert ^*\Vert _1\frac{\eta _1(n)}{\delta }(\sqrt{\frac{\log (d)}{n\delta }} + \delta ^{\ell })+ o_{}(1)\\=&o_{}(1).\end{aligned}\) 
Similarly, since \(\nabla R(^*) = 0\) , imply that
\(\begin{aligned}[b]&I_2 = (n\delta )^{1/2}|(\widehat{}^{(1)} - ^{*})(\nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - \nabla R(^*))|\\\le &(n\delta )^{1/2}\Vert \widehat{}^{(1)} - ^*\Vert _1\Vert \nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - \nabla R(^*)\Vert _\infty \\\le &(n\delta )^{1/2}\Vert \widehat{}^{(1)} - ^*\Vert _1\bigg (\Vert \nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - \nabla R_\delta (\widehat{}_0^{(2)})\Vert _\infty \\ &~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + \Vert \nabla R_\delta (\widehat{}_0^{(2)}) - \nabla R(\widehat{}_0^{(2)})\Vert _\infty +\Vert \nabla R(\widehat{}_0^{(2)}) - \nabla R(^*)\Vert _\infty \bigg ).\end{aligned}\) 
Since \(\widehat{}_0^{(2)}\)  depends on the set of samples that is disjoint with \(_1\) , Lemma  together with Lemma  implies that
\(\Vert \nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - \nabla R_\delta (\widehat{}_0^{(2)})\Vert _\infty = _{}(\sqrt{\frac{\log (d)}{n\delta }}).\) 
This in combine with Lemma  and  further implies that
\(I_2 \lesssim (n\delta )^{1/2}\Vert ^*\Vert _1\eta _2(n)\bigg (M_n \eta _1(n) + \sqrt{\frac{\log (d)}{n\delta }} + \delta ^{\ell }\bigg ) = o_{}(1). \) 
Putting all above together with the results from Lemma  and Lemma  as well as the Slutsky's theorem, we obtain that the bias corrected decorrelated score statistic \(\widehat{U}_n\stackrel{d}{\rightarrow } N(0,1)\)  under the null hypothesis.

Proof of Theorem  REF
Recall that \(\widehat{S}_\delta (0,\widehat{}) = \frac{1}{2} \big ( \widehat{S}^{(1)}_\delta (0,\widehat{}^{(2)}) + \widehat{S}^{(2)}_\delta (0,\widehat{}^{(1)}) \big )\) . Let's focus on \(\widehat{S}^{(1)}_\delta (0,\widehat{}^{(2)}) \) . By definition
\(\begin{aligned}[b]&\widehat{S}^{(1)}(0,\widehat{}^{(2)})= \widehat{}^{(1)T}\nabla R_\delta ^{n_{(1)}}(0,\widehat{}^{(2)})\\=&\underbrace{(\widehat{}^{(1)} - ^*)^T\nabla R_\delta ^{n_{(1)}}(0,\widehat{}^{(2)})}_{I_1} + ^{*T}\nabla R_\delta ^{n_{(1)}}(0,\widehat{}^{(2)})\\=&I_1 + \underbrace{^{*T}\big (\nabla R_\delta ^{n_{(1)}}(0,\widehat{}^{(2)}) - \nabla R_\delta ^{n_{(1)}}(0,^*)\big )}_{I_2} + ^{*T}\nabla R_\delta ^{n_{(1)}}(0,^*) \\=&I_1 + I_2 + ^{*T}\nabla R_\delta ^{n_{(1)}}(^*) - \theta ^*^{*T}\nabla ^2_{\cdot \theta } R(^*) + \underbrace{^{*T}(\nabla R_\delta ^{n_{(1)}}(0,^*) - \nabla R_\delta ^{n_{(1)}}(^*)) + \theta ^*^{*T}\nabla ^2_{\cdot \theta }R(^*)}_{I_3}\\=&S^{(1)}(^*)- \theta ^*^{*T}\nabla ^2_{\cdot \theta }R(^*) + I_1 + I_2 + I_3.\end{aligned}\) 
For \(I_1\) , we know
\(I_1 \le \Vert \widehat{}^{(1)} - ^*\Vert _1\Vert \nabla R_\delta ^n(0,\widehat{})\Vert _\infty = _{}(\Vert ^*\Vert _1\eta _2(n)(\sqrt{\log (d)/(n\delta )} + \delta ^{\ell } + M_n(\theta ^* + \eta _1(n)))) = o_{}((n\delta )^{-1/2}).\) 
Following a similar proof of Lemma , we can show that
\(I_2 = _{}\big (\Vert ^*\Vert _1\eta _1(n)\big [\delta ^{\ell -1} + \sqrt{\log (d)/(n\delta ^3)} + M_n\theta ^*\big ]\big ) = o_{}((n\delta )^{-1/2}).\) 
Finally, for \(I_3\) , since by definition
\(\begin{aligned}[b]^{*T}\nabla ^2_{\cdot \theta }R(^*) = \nabla ^2_{\theta |}R(^*),\end{aligned}\) 
Lemma  implies that \(I_3 = o_{}((n\delta )^{-1/2})\) .
Put all pieces together, we have shown that
\(|\widehat{S}^{(1)}(0,\widehat{}^{(2)}) - (S^{(1)}(^*)- \theta ^*\nabla ^2_{\theta |}R(^*))| = o_{}((n\delta )^{-1/2}).\) 
A similar result will also hold for \(\widehat{S}^{(2)}(0,\widehat{}^{(1)})\) , and thus we conclude
\(\begin{aligned}[b]|\widehat{S}(0,\widehat{}) - (S(^*)- \theta ^*\nabla ^2_{\theta |}R(^*))| = o_{}((n\delta )^{-1/2}).\end{aligned}\) 
At this point the conclusion of (REF ) is shown. To show (REF ), the above formula also implies that
\(\begin{aligned}[b](|(n\delta )^{1/2}\widehat{S}(0,\widehat{})/\sigma ^*| \le t) \le (L(n) \le (n\delta )^{1/2}S(^*)/\sigma ^* \le U(n)) + o(1),\end{aligned}\) 
where \(L(n) = -t - q(n) + \tilde{C}\sigma ^{*-1}\nabla ^2_{\theta |}R(^*)n^{-\phi }(n\delta )^{1/2}\) , \(U(n) = t + q(n) + \tilde{C}\sigma ^{*-1}\nabla ^2_{\theta |}R(^*)n^{-\phi }(n\delta )^{1/2}\) , and \(q(n) = o(1)\)  is some deterministic sequence. Since \(n^{-\phi }(n\delta )^{1/2}\nabla ^2_{\theta |}R(^*)\sigma ^{*-1}\rightarrow \infty \) , it is easily seen that \((|(n\delta )^{1/2}\widehat{S}(0,\widehat{})/\sigma ^*| \le t) \rightarrow 0\) . Since \(\widehat{\sigma }\)  is consistent, for \(n\)  large enough we will get \(|\widehat{\sigma }/\sigma - 1|\le 3\) , which finally implies the desired result.
This completes the proof.

Proof of Lemma 
It suffices to show the rate for \(\widehat{}^{(1)T}\widehat{}^{(1)}(\delta )\widehat{}^{(1)}\) .
Let \(\tilde{}(\delta ) = \bigg [\nabla \bar{R}_\delta ^1(\widehat{}^{(2)})\nabla \bar{R}_\delta ^1(\widehat{}^{(2)})^T\bigg ]\) , and \((\delta )=[\nabla \bar{R}^1_\delta (^*)\nabla \bar{R}^1_\delta (^*)^T]\) . We firstly look at
\(\begin{aligned}[b]\Vert \widehat{}^{(1)}(\delta ) - (\delta )\Vert _{\max } \le \Vert \widehat{}^{(1)}(\delta ) - \tilde{}(\delta )\Vert _{\max } + \Vert \tilde{}(\delta ) - (\delta )\Vert _{\max }.\end{aligned}\) 
Now we bound the first term. For any \(\delta \in \Delta \) , we can always find a \(\widetilde{\delta }\in \widetilde{\Delta }\)  such that \(|\delta - \widetilde{\delta }| \lesssim n^{-\rho }\) , where \(\rho \)  is some constant that can be arbitrarily large and the set \(\widetilde{\Delta }\)  that is a subset of \(\Delta \)  has cardinality of order \((n^{\rho })\) . For such pair \((\delta ,\widetilde{\delta })\) , we can show that
\(\begin{aligned}[b]&\sqrt{n\delta ^3}\Vert \widehat{}^{(1)}(\delta ) - \tilde{}(\delta )\Vert _{\max }\\ \lesssim & \sqrt{n\widetilde{\delta }^3}\bigg (\Vert \widehat{}^{(1)}(\widetilde{\delta }) - \tilde{}(\widetilde{\delta })\Vert _{\max } + \Vert \widehat{}^{(1)}(\delta ) - \widehat{}^{(1)}(\tilde{\delta })\Vert _{\max } + \Vert \tilde{}(\widetilde{\delta }) - \tilde{}(\delta )\Vert _{\max }\bigg )\\\lesssim & \sqrt{n\widetilde{\delta }^3}\Vert \widehat{}^{(1)}(\widetilde{\delta }) - \tilde{}(\widetilde{\delta })\Vert _{\max } + (n^{-1/2}),\end{aligned}\) 
where the first inequality is by taking \(\rho \)  large enough.
To see last step of above, recall that under Assumptions REF , REF  and  REF
\(\begin{aligned}[b]\Vert \widehat{}^{(1)}(\delta ) - \widehat{}^{(1)}(\widetilde{\delta })\Vert _{\max }=& \max _{j,k}\bigg |\frac{1}{|_1|}\sum _{i\in _1}w(y_i)^2_{ij}_{ik} \bigg [\frac{K^2((x_i - \widehat{}^{(2)T}_i)/\delta )}{\delta ^2}- \frac{K^2((x_i - \widehat{}^{(2)T}_i)/\widetilde{\delta })}{\widetilde{\delta }^2}\bigg ]\bigg |\\\lesssim & M_n^2(\frac{1}{\delta ^2} - \frac{1}{\widetilde{\delta }^2}).\end{aligned}\) 
Thus, taking \(\rho \)  large enough will ensure \(\sqrt{n\widetilde{\delta }^3}(1+o(1))\Vert \widehat{}^{(1)}(\delta ) - \widehat{}^{(1)}(\widetilde{\delta })\Vert _{\max } = (n^{-1/2})\) . With a similar derivation we will also obtain \(\sqrt{n\widetilde{\delta }^3}(1+o(1))\Vert \widetilde{}(\delta ) - \widetilde{}(\widetilde{\delta })\Vert _{\max } = (n^{-1/2})\) , and thus (REF ) holds.
Now we start to bound \(\max _{\widetilde{\delta }\in \widetilde{\Delta }}\sqrt{n\widetilde{\delta }^3}\Vert \widehat{}^{(1)}(\widetilde{\delta }) - \tilde{}(\widetilde{\delta })\Vert _{\max }\) .
Since
\(|\widehat{}^{(1)}_{ijk}(\widetilde{\delta })| = |(\nabla \bar{R}_{\widetilde{\delta }}^i(\widehat{}^{(2)})\nabla \bar{R}_{\widetilde{\delta }}^i(\widehat{}^{(2)})^T)_{jk}| \lesssim \frac{M_n^2}{\widetilde{\delta }^2}\) 
and
\(Var[(\nabla \bar{R}_{\widetilde{\delta }}^i(\widehat{}^{(2)})\nabla \bar{R}_{\widetilde{\delta }}^i(\widehat{}^{(2)})^T)_{jk}] \lesssim \frac{1}{\widetilde{\delta }^3},\) 
by applying Bernstein inequality similar to the proof of Lemma  , conditioned on \(\widehat{}^{(2)}\)  we can show that
\(\begin{aligned}[b]&_{\widehat{}^{(2)}}(\max _{\widetilde{\delta }\in \widetilde{\Delta }}\sqrt{n\widetilde{\delta }^3}\Vert \widehat{}^{(1)}(\widetilde{\delta }) - \tilde{}(\widetilde{\delta })\Vert _{\max } > t)\\\le &\sum _{\widetilde{\delta }\in \widetilde{\Delta }}\sum _{j = 1}^d\sum _{k = 1}^d_{\widehat{}^{(2)}}(\sqrt{n\widetilde{\delta }^3}|\widehat{}^{(1)}(\widetilde{\delta }) - \tilde{}(\widetilde{\delta })|_{jk} > t)\\\lesssim &n^{\rho }d^2\exp \bigg (-\frac{\frac{1}{2}t^2/\widetilde{\delta }^3}{\frac{c_0}{\widetilde{\delta }^3} + \frac{c1M_n^2t}{3\widetilde{\delta }^3\sqrt{n\widetilde{\delta }}} } \bigg ),\end{aligned}\) 
where \(c_0,c_1>0\)  are some constants. By taking \(t = c_2\sqrt{\log (nd)}\)  for some constant \(c_2\)  large enough, we obtain that conditioned on \(\widehat{}^{(2)}\) , with probability greater than \(1 - ((nd)^{-1})\)
\(\begin{aligned}[b]\Vert \widehat{}^{(1)}(\widetilde{\delta }) - \tilde{}(\widetilde{\delta })\Vert _{\max }\lesssim \sqrt{\frac{\log (nd)}{n\widetilde{\delta }^3}}\end{aligned}\) 
uniformly over \(\widetilde{\delta }\in \widetilde{\Delta }\) .
This combines with Lemma  further implies that with probability approaching to 1
\(\begin{aligned}[b]\Vert \widehat{}^{(1)}(\delta ) - \tilde{}(\delta )\Vert _{\max } \lesssim \sqrt{\frac{\log (nd)}{n\delta ^3}}\end{aligned}\) 
uniformly over \(\delta \in \Delta \) .
For the second term on the RHS of (REF ), similar to Lemma , we can show that
\(\begin{aligned}[b]\Vert \tilde{}(\delta ) - (\delta )\Vert _{\max } \lesssim \frac{M_n\eta _1(n)}{\delta }.\end{aligned}\) 
This implies that
\(\begin{aligned}[b]\Vert \widehat{}(\delta ) - (\delta )\Vert _{\max } = \frac{1}{\delta }_{}(\sqrt{\frac{\log (nd)}{n\delta }} \vee M_n\eta _1(n)).\end{aligned}\) 
Meanwhile, since
\(\begin{aligned}[b]\Vert \delta \cdot (\delta )\Vert _{\max } =& \max _{i,j}\bigg |\sum _{y}w(y)^2\int \frac{z_iz_j}{\delta } \int K^2(\frac{y(x - ^{*T})}{\delta })f(x|y,)dxf(y,)d\bigg |\\=&(1),\end{aligned}\) 
under the condition that \(\sqrt{\frac{\log (nd)}{n\delta }} \vee M_n\eta _1(n) = o(1)\) , we know \(\Vert \delta \widehat{}(\delta )\Vert _\infty = _{}(1)\) .
Finally, by triangle inequality
\(\begin{aligned}[b]&|\widehat{}^{(1)T}\widehat{}^{(1)}(\delta )\widehat{}^{(1)} - ^{*T}{}(\delta )^*|\\\le & \Vert \widehat{}^{(1)} - ^*\Vert _1\Vert \widehat{}^{(1)}(\delta )(\widehat{}^{(1)} - )\Vert _\infty + 2\Vert ^{*T}\widehat{}^{(1)}(\delta )\Vert _\infty \Vert \widehat{}^{(1)} - ^*\Vert _1 + |^{*T}(\widehat{}^{(1)}(\delta ) - {}(\delta ))^*|,\end{aligned}\) 
which further implies that with probability approaching 1
\(|\widehat{}^{(1)T}\widehat{}^{(1)}(\delta )\widehat{}^{(1)} - ^{*T}{}(\delta )^*| \lesssim \frac{\Vert ^*\Vert _1^2}{\delta }\bigg (\eta _2(n) + \sqrt{\frac{\log (nd)}{n\delta }} + M_n\eta _1(n) \bigg )\) 
uniformly over \(\delta \in \Delta \) . This completes the proof.

Proof of Lemma 
It suffices to bound \(\bigg |\bigg (\widehat{}^{(1)T} \frac{1}{|_1|}\sum _{i \in _1}A_i(\widehat{}^{(2)},\delta )\bigg )^2 - SB(\delta )\bigg |\) . Define
\(\begin{aligned}[b]\widetilde{A}(,\delta ) =& _{}(A_1(,\delta ))\\=&\sum _{y}w(y)y\int _{}\int _t J(t)\int _uK(u)[f(u\delta +tb+^{T}|y,) - f(tb+ ^T|y,)]dudtf(y,)d.\end{aligned}\) 
By definition,
\(\begin{aligned}[b]&\bigg |\bigg (\widehat{}^{(1)T} \frac{1}{|_1|}\sum _{i \in _1}A_i(\widehat{}^{(2)},\delta )\bigg )^2 - SB(\delta )\bigg |\\=& \bigg |\bigg (\widehat{}^{(1)T} \frac{1}{|_1|}\sum _{i \in _1}A_i(\widehat{}^{(2)},\delta )\bigg )^2 - \big (^{*T}A(^*,\delta )\big )^2\bigg |\\\le & \bigg |\bigg (\widehat{}^{(1)T} \frac{1}{|_1|}\sum _{i \in _1}A_i(\widehat{}^{(2)},\delta )\bigg )^2 - \big (\widehat{}^{(1)T}A(\widehat{}^{(2)},\delta )\big )^2\bigg | + \bigg |\big (\widehat{}^{(1)T}A(\widehat{}^{(2)},\delta )\big )^2- \big (^{*T}A(^*,\delta )\big )^2\bigg |\\\le &\underbrace{\bigg |\bigg (\widehat{}^{(1)T}\big (\frac{1}{|_1|}\sum _{i\in _1}A_i(\widehat{}^{(2)},\delta ) - \widetilde{A}(\hat{}^{(2)},\delta )\big )\bigg )^2\bigg |}_{I_1}\\&~~~~~~~~ + 2\underbrace{\bigg |\bigg (\widehat{}^{(1)T}\widetilde{A}(\hat{}^{(2)},\delta )\cdot \widehat{}^{(1)T}\big (\frac{1}{|_1|}\sum _{i\in _1}A_i(\widehat{}^{(2)},\delta ) - \widetilde{A}(\hat{}^{(2)},\delta )\big )\bigg |}_{I_2}\\&~~~~~~~~ + \underbrace{\bigg |\big (\widehat{}^{(1)T}\widetilde{A}(\hat{}^{(2)},\delta )\big )^2 - \big (\widehat{}^{(1)T}A(\widehat{}^{(2)},\delta )\big )^2\bigg |}_{I_3} + \underbrace{\bigg |\big (\widehat{}^{(1)T}A(\widehat{}^{(2)},\delta )\big )^2- \big (^{*T}A(^*,\delta )\big )^2\bigg |}_{I_4}.\end{aligned}\) 
Now we start to bound each term.

For \(I_1\) ,
we firstly study \(\delta ^{-\ell }\Vert \frac{1}{|_1|}\sum _{i\in _1}A_i(\widehat{}^{(2)},\delta ) - \widetilde{A}(\hat{}^{(2)},\delta )\Vert _\infty \)  condition on \(\widehat{}^{(2)}\) .
Similar to the proof of Lemma , it suffices to bound
\(\begin{aligned}[b]\widetilde{\delta }^{-\ell }\Vert \frac{1}{|_1|}\sum _{i\in _1}A_i(\widehat{}^{(2)},\widetilde{\delta }) - \widetilde{A}(\hat{}^{(2)},\widetilde{\delta })\Vert _\infty \end{aligned}\) 
uniformly over \(\widetilde{\delta }\in \widetilde{\Delta }\) , where \(\widetilde{\Delta }\)  is a subset of \(\Delta \)  with cardinality of order \((n^\rho )\)  for some constant \(\rho >0\) , such that for each \(\delta \in \Delta \) , there exist \(\widetilde{\delta }\in \widetilde{\Delta }\)  with \(|\delta - \widetilde{\delta }| \lesssim n^{-\rho }\) .
Let \(T_{ij}(\widetilde{\delta }) = \widetilde{\delta }^{-\ell }\big ( A_i(\widehat{}^{(2)},\widetilde{\delta }) - \widetilde{A}(\hat{}^{(2)},\widetilde{\delta }) \big )_j\) , we know \(_{\widehat{}^{(2)}} T_{ij}(\widetilde{\delta }) = 0\) . Moreover for all \(j\) ,
\(\begin{aligned}[b]\widetilde{\delta }^{-\ell }|(A_{i}(\widehat{}^{(2)},\widetilde{\delta }))_j| =& \widetilde{\delta }^{-\ell }\bigg |w(y_i)\frac{z_{ij}y_i}{b}\int K(u) \big [J(\frac{x_i - \widehat{}^{(2)T}_i - u\widetilde{\delta }}{b}) - J(\frac{x_i - \widehat{}^{(2)T}_i}{b})\big ]\biggr \rbrace  du\bigg | \\=& \widetilde{\delta }^{-\ell }\bigg |w(y_i)\frac{z_{ij}y_i}{b}\int K(u) \frac{\widetilde{\delta }^\ell }{b^{\ell }}\frac{u^\ell }{\ell !}J^{\ell }(\frac{x_i - \widehat{}^{(2)T}_i - \tau u\widetilde{\delta }}{b})\big ]\biggr \rbrace  du\bigg | \\\lesssim & \frac{M_n}{b^{\ell +1}},\end{aligned}\) 
where \(\tau \in [0,1]\)  and the second equality is because \(K\)  is of order \(\ell \)  and mean value theorem. This implies that \(|T_{ij}(\widetilde{\delta })|\lesssim \frac{M_n}{b^{\ell +1}}\) . Now we look at \(_{\widehat{}^{(2)}}[T^2_{ij}(\widetilde{\delta })]\) .
Notice that for all \(j\)
\(\begin{aligned}[b]\widetilde{A}(\hat{}^{(2)},\widetilde{\delta })_j =& \sum _{y}w(y)\int _{}_j y\int _t J(t)\int _uK(u)[f(u\widetilde{\delta }+tb+\widehat{}^{(2)T}|y,) - f(tb+\widehat{}^{(2)T}|y,)]dudtf(y,)d\\=&\sum _{y}w(y)\int _{}_j y\int _t J(t)\int _uK(u)\frac{(u\widetilde{\delta })^{\ell }}{\ell !} f^{(\ell )}(\tau u\widetilde{\delta }+tb+\widehat{}^{(2)T}|y,)dudtf(y,) d\\\lesssim &\widetilde{\delta }^{\ell },\end{aligned}\) 
where the second equality is because \(K\)  is of order \(\ell \)  and \(\tau \in [0,1]\) .
For the second moment, we have
\(\begin{aligned}[b]&_{\widehat{}^{(2)}}[(\widetilde{\delta }^{-\ell }A_{1}(\widehat{}^{(2)},\widetilde{\delta }))_j^2]\\=&\widetilde{\delta }^{-2\ell }_{\widehat{}^{(2)}} \bigg [w(y)^2 \frac{z_j^2}{b^2}\bigg (\int K(u) \big [J(\frac{x - \widehat{}^{(2)T}- u\widetilde{\delta }}{b}) - J(\frac{x - \widehat{}^{(2)T}}{b})\big ]\biggr \rbrace  du\bigg )^2\bigg ] \\=&\widetilde{\delta }^{-2\ell }_{\widehat{}^{(2)}} \bigg [w(y)^2 \frac{z_j^2}{b^2}\bigg (\int K(u) \frac{\widetilde{\delta }^\ell }{b^{\ell }}\frac{u^\ell }{\ell !}J^{\ell }(\frac{x - \widehat{}^{(2)T}- \tau u\widetilde{\delta }}{b})du\bigg )^2\bigg ] \\=&\frac{1}{b^{2\ell +1}} \bigg | \sum _{y}\int _z w(y)^2z_j^2\int _t \bigg (\int K(u) \frac{u^\ell }{\ell !}J^{\ell }(t - \frac{\tau u\widetilde{\delta }}{b})d u\bigg )^2f(tb + \widehat{}^{(2)T}|y,)dt f(y,) d\bigg | \\\lesssim & \frac{1}{b^{2\ell +1}},\end{aligned}\) 
where the last equality is due to a change of variable \(t = \frac{x - \widehat{}^{(2)T}}{b}\) .
Since \(b\rightarrow 0\) , we conclude that \(_{\widehat{}^{(2)}}[T^2_{ij}(\widetilde{\delta })]\lesssim \frac{1}{b^{2\ell +1}}.\)  Now applying Bernstein inequality similar to Lemma  with \(M_n\sqrt{\frac{\log (d)}{nb}} = (1)\) , second moment of order \(\frac{1}{b^{2\ell +1}}\)  and each term is bounded by \(\frac{M_n}{b^{\ell +1}}\) , we will obtain that with probability greater than \(1 - ((nd)^{-1})\) , conditioned on \(\widehat{}^{(2)}\)
\(\begin{aligned}[b]\Vert \frac{1}{|_1|}\sum _{i\in _1}A_i(\widehat{}^{(2)},\widetilde{\delta }) - A_i(\widehat{}^{(2)},\widetilde{\delta })\Vert _\infty \lesssim \frac{\widetilde{\delta }^{\ell }}{b^{\ell }}\sqrt{\frac{\log (nd)}{nb}},\end{aligned}\) 
which together with Lemma  implies that with probability approaching 1
\(\begin{aligned}[b]\Vert \frac{1}{|_1|}\sum _{i\in _1}A_i(\widehat{}^{(2)},\delta ) - _{\widehat{}^{(2)}} A_i(\widehat{}^{(2)},\delta )\Vert _\infty \lesssim \frac{\delta ^{\ell }}{b^{\ell }}\sqrt{\frac{\log (nd)}{nb}}\end{aligned}\) 
holds uniformly for all \(\delta \in \Delta \) .
Therefore we conclude that
\(\begin{aligned}[b]I_1 = _{}(\Vert ^*\Vert _1^2\frac{\delta ^{2\ell }}{b^{2\ell }}\frac{\log (nd)}{nb}).\end{aligned}\) 

For \(I_2\) , (REF ) implies that
\(\Vert _{\widehat{}^{(2)}} A_1(\widehat{}^{(2)},\delta )\Vert _{\infty } \lesssim \delta ^{\ell }\)  regardless the choice of \(\widehat{}^{(2)}\) . This combined with (REF ) further implies that
\(\begin{aligned}[b]I_2 = _{}(\Vert ^*\Vert _1^2 \delta ^{\ell }\sqrt{\frac{\delta ^{2\ell }}{b^{2\ell }}\frac{\log (nd)}{nb}}) = _{}(\Vert ^*\Vert _1^2\frac{\delta ^{2\ell }}{b^{\ell }}\sqrt{\frac{\log (nd)}{nb}}).\end{aligned}\) 

For \(I_3\) , by definition for all \(j\)  we can write
\(\begin{aligned}[b]&\big ( \widetilde{A}(\widehat{}^{(2)},\delta ) - A(\widehat{}^{(2)},\delta )\big )_j\\=& \sum _{y}w(y)\int _{}_j y\int _t J(t)\int _uK(u)[(f(u\delta +tb+\widehat{}^{(2)T}|y,) - f(tb+\widehat{}^{(2)T}|y,))\\&~~~~~~~~ - (f(u\delta + \widehat{}^{(2)T}|y,) - f(\widehat{}^{(2)T}|y,)) ]dudtf(y,)d\\=&\sum _{y}w(y)\int _{}_j y\int _u\sum _{j = \ell }^{\ell + r -1}\frac{(u\delta )^j}{j!}K(u)\int _t J(t) [ f^{(j)}(tb+\widehat{}^{(2)T}|y,) - f^{(j)}(\widehat{}^{(2)T}|y,)]dtduf(y,)d\\&~~~~~~~~ + (\delta ^{\ell + r})\\=& \sum _{y}w(y)\int _{}_j y\int _u\sum _{j = \ell }^{\ell + r - 1}\frac{(u\delta )^j}{j!}K(u)\int _t J(t) \frac{(tb)^{\ell + r - j}}{(\ell + r - j)!} f^{(\ell + r)}(\tau tb+\widehat{}^{(2)T}|y,) dtduf(y,)d+ (\delta ^{\ell + r})\\=&(\delta ^{\ell }(\delta \vee b)^{r}),\end{aligned}\) 
where \(\tau \in [0,1]\) .
Here the first equality is by definition, the second equality uses the property that \(\int K(u)u^i du = 0~\forall ~i < \ell \) , and the third equality uses the property that \(\int J(u)u^i du = 0~\forall ~ i < r\)  and the mean value theorem. This implies that
\(\begin{aligned}[b]\Vert \widetilde{A}(\widehat{}^{(2)},\delta ) - A(\widehat{}^{(2)},\delta )\Vert _\infty = (\delta ^{\ell }(\delta \vee b)^{r}).\end{aligned}\) 
Meanwhile, by (REF ) and a similar derivation, we know both \(\Vert \widetilde{A}(\widehat{}^{(2)},\delta )\Vert _{\infty }\)  and \(\Vert A(\widehat{}^{(2)},\delta )\Vert _{\infty }\)  are of order \(\delta ^{\ell }\) . These together imply that
\(\begin{aligned}[b]I_3 = (\Vert ^*\Vert _1^2\delta ^{2\ell }(\delta \vee b)^{r}).\end{aligned}\) 

For \(I_4\) , again \(\Vert A(\widehat{}^{(2)},\delta )\Vert _\infty \asymp \Vert A(^*,\delta )\Vert _\infty = (\delta ^\ell )\) . For the difference, we have
\(\begin{aligned}[b]&\Vert A(\widehat{}^{(2)},\delta ) - A(^*,\delta )\Vert _\infty \\=&\max _j\bigg |\int _u K(u) \bigg [\sum _y w(y)\int _{} z_j y \big (f(u\delta + \widehat{}^{(2)T}|y,) - f(\widehat{}^{(2)T}|y,)\\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - f(u\delta + ^{*T}|y,) + f(^{*T}|y,)\big ) f(y,) d\bigg ] du \bigg |\\=&\max _j\bigg | \sum _{y} w(y)\int _{}z_j y \int _{u} K(u)\bigg [\big (f(u\delta + \widehat{}^{(2)T}|y,) - f(u\delta + ^{*T}|y,)\big )\\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - \big (f(\widehat{}^{(2)T}|y,) - f(^{*T}|y,)\big )\bigg ] f(y,) ddu\bigg | \\=&\max _j\bigg | \sum _{y} w(y)\int _{}z_j y \int _{u} K(u)\bigg [(\widehat{} - ^*)^T\big (f^{\prime }(u\delta + ^{*T}|y,)- f^{\prime }(^{*T}|y,)\big ) \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + ((\widehat{} - ^*)^T)^2\big (f^{\prime \prime }(u\delta + \tilde{}^T) - f^{\prime \prime }( \breve{}^T)\big )\bigg ]duf(y,)d\bigg | \\\lesssim &M_n\Vert \widehat{} - ^*\Vert _1\delta ^{\ell } + M_n^2\Vert \widehat{} - ^*\Vert _1^2,\end{aligned}\) 
where the last step follows from an \(\ell \)  order Taylor expansion of \(f^{\prime }(u\delta + ^{*T}|y,)- f^{\prime }(^{*T}|y,)\)  for the first term and boundedness of \(f^{\prime \prime }\)  for the second term.
This implies that
\(\begin{aligned}[b]\Vert A(\widehat{}^{(2)},\delta ) - A(^*,\delta )\Vert _\infty \lesssim M_n\eta _1(n)(\delta ^{\ell } + M_n\eta _1(n))\end{aligned}\) 
with probability approaching 1 by Assumption REF . Therefore, we can obtain
\(\begin{aligned}[b]I_4 \le & \bigg |(^{(1)T}A(\widehat{}^{(2)},\delta ) - ^{*T}A(^*,\delta ))(^{(1)T}A(\widehat{}^{(2)},\delta ) + ^{*T}A(^*,\delta ))\bigg |\\\le &\bigg |^{(1)T}(A(\widehat{}^{(2)},\delta ) - A(^*,\delta )) + (^{(1)T}- ^{*T})A(^*,\delta )\bigg |\bigg |^{(1)T}A(\widehat{}^{(2)},\delta ) + ^{*T}A(^*,\delta )\bigg |\\=&\Vert ^*\Vert _1^2\delta ^{2\ell }_{}(M_n\eta _1(n)(1 \vee M_n\eta _1(n)/\delta ^{\ell }) + \eta _2(n)).\end{aligned}\) 

Combining all bounds for \(I_1,\cdots ,I_4\) , we know that uniformly over all \(\delta \in \Delta \)
\(\begin{aligned}[b]&\bigg |\bigg (\widehat{}^{(1)T} \frac{1}{|_1|}\sum _{i \in _1}A_i(\widehat{}^{(2)},\delta )\bigg )^2 - SB(\delta )\bigg |\\\lesssim &\Vert ^*\Vert _1^2\delta ^{2\ell } \bigg (\frac{\log (nd)}{nb^{2\ell +1}} + \sqrt{\frac{\log (nd)}{nb^{2\ell +1}}} + (\delta \vee b)^r + M_n \eta _1(n)(1 \vee \frac{M_n\eta _1(n)}{\delta ^{\ell }}) + \eta _2(n)\bigg ).\end{aligned}\) 
This completes the proof.

Proof of Theorem 
By Lemma  and Lemma , uniformly over all \(\delta \in \Delta \)
\(\begin{aligned}[b]&|\widehat{M}(\delta ) - M(\delta )|\\=&\Vert ^*\Vert _1^2\frac{1}{n\delta }_{}\bigg (\eta _2(n) + \sqrt{\frac{\log (nd)}{n\delta }} + M_n\eta _1(n) \bigg ) \\&~~~~~~ + \Vert ^*\Vert _1^2\delta ^{2\ell } _{}\bigg (\frac{\log (nd)}{nb^{2\ell +1}} + \sqrt{\frac{\log (nd)}{nb^{2\ell +1}}} + (\delta \vee b)^r + M_n \eta _1(n)(1 \vee \frac{M_n\eta _1(n)}{\delta ^{\ell }}) + \eta _2(n)\bigg )\\\lesssim & \frac{1}{n\delta }\psi _1(n) + \delta ^{2\ell }\psi _2(n).\end{aligned}\) 
Recall that \(M(\delta ) = \frac{1}{n} V(\delta ) + \frac{n-1}{n}SB(\delta ),\)  where
\(\begin{aligned}[b]V(\delta ) = \frac{1}{\delta }(^{*T}^*^*)(1+o(1))~~~~~\text{and}~~ {SB(\delta )} = \delta ^{2\ell }(^{*T}^*)^2(1+o(1)).\end{aligned}\) 
This implies that \(\delta ^*=(\frac{1}{n}\frac{^{*T}^*^*}{2\ell (^{*T}^*)^2})^{1/(2\ell + 1)}(1+o(1))\) . By Assumption REF  we know \(M(\delta ) \gtrsim \frac{1}{n\delta } + \delta ^{2\ell }\) , which implies
\(\frac{|\widehat{M}(\delta ) - M(\delta )|}{M(\delta )} \lesssim \psi _1(n) \vee \psi _2(n)= o_{}(1),\) 
uniformly over all \(\delta \in \Delta \) . That means for any \(\epsilon >0\) , uniformly over all \(\delta \in \Delta \) , the event \((1-\epsilon )M(\delta )\le \widehat{M}(\delta )\le (1+\epsilon )M(\delta )\)  holds with probability tending to 1. Under this event, we have that
\(\begin{aligned}[b]M(\hat{\delta })&=\widehat{M}(\hat{\delta })+[M(\hat{\delta })-\hat{M}(\hat{\delta })]\\&\le \widehat{M}(\delta ^*)+[M(\hat{\delta })-\hat{M}(\hat{\delta })]\\&\le (1+\epsilon ) M(\delta ^*)+\epsilon M(\hat{\delta }),\end{aligned}\) 
where the first inequality holds because \(\hat{\delta }\)  is the minimizer of \(\widehat{M}(\delta )\)  and the second inequality follows from \(\widehat{M}(\delta ^*)\le (1+\epsilon )M(\delta ^*)\)  and \(\widehat{M}(\widehat{\delta })\ge (1-\epsilon )M(\widehat{\delta })\) . As a result, we obtain \(M(\hat{\delta })\le \frac{1+\epsilon }{1-\epsilon }M(\delta ^*)\) , from which we can claim that \(\hat{\delta }\le \frac{3}{2}\delta ^*\) . To see this, let us consider the complement case \(\hat{\delta }> \frac{3}{2}\delta ^*\) . When it holds, we know from (REF ) that
\(M(\hat{\delta })\ge (\frac{3}{2})^{2\ell }SB(\delta ^*)(1+o(1))=(\frac{3}{2})^{2\ell }\frac{1}{2\ell +1}M(\delta ^*)(1+o(1))\ge 1.01M(\delta ^*)(1+o(1)),\) 
where the second step follows from \(M(\delta ^*)=(2\ell +1)SB(\delta ^*)(1+o(1))\)  and last step holds by the condition \(\ell \ge 2\) . However, the above inequality contradicts with \(M(\hat{\delta })\le \frac{1+\epsilon }{1-\epsilon }M(\delta ^*)\)  for some sufficiently small \(\epsilon \) . This justifies the statement \(\hat{\delta }\le \frac{3}{2}\delta ^*\) . Following a similar argument, we can show that \(\hat{\delta }\ge \frac{1}{2}\delta ^*\) . Thus, with probability tending to 1, we have \(|\hat{\delta }-\delta ^*|/\delta ^*\le 1/2\) .
By definition, we have \(M^{\prime }(\delta ^*) = 0\)  and \(\widehat{M}^{\prime }(\widehat{\delta }) = 0\) . This implies that
\(\begin{aligned}[b]\widehat{M}^{\prime }(\widehat{\delta }) - M^{\prime }(\widehat{\delta }) = -M^{\prime \prime }(\tilde{\delta })(\widehat{\delta } - \delta ^*),\end{aligned}\) 
for some intermediate value \(\tilde{\delta }\) , which gives
\(\begin{aligned}[b](\widehat{\delta } - \delta ^*) = \frac{\widehat{M}^{\prime }(\widehat{\delta }) - M^{\prime }(\widehat{\delta }) }{-M^{\prime \prime }(\tilde{\delta })}.\end{aligned}\) 
For any \({\delta } \asymp \delta ^*\) , after some algebra, we can show that
\(\begin{aligned}[b]M^{\prime \prime }(\delta ) \asymp \frac{1}{\delta ^{3}n}.\end{aligned}\) 
Similar to the proof of (REF ), we can show that \((\widehat{M}^{\prime } - M^{\prime })(\delta )\)  has the same order as \((\widehat{M} - M)(\delta )\) , except for an additional factor \(\frac{1}{\delta }\) . Since \(|\hat{\delta }-\delta ^*|/\delta ^*\le 1/2\)  holds in probability, it implies \(\widehat{\delta }\)  and \(\tilde{\delta }\)  are of the same order of \(\delta ^*\) . Thus, (REF ) implies
\(\begin{aligned}[b]\frac{\widehat{\delta } - \delta ^*}{\delta ^*} \lesssim & \psi _1(n) \vee \psi _2(n).\end{aligned}\) 
This completes the proof.

Proof of Theorem  
The proof is very similar to the proof of Theorem REF  so we only give a sketch here. Following the same derivation of Lemma REF , we can show that
\(\begin{aligned}[b](n\delta )^{1/2}\frac{S_\delta ^L(0,^*,^*_L) - \delta ^{\ell }\mu ^*_L}{\sigma ^*_L} \stackrel{d}{\rightarrow } N(0,1).\end{aligned}\) 
Next, following the derivation of Theorem REF , we can show that \(I_1 = (n\delta )^{1/2}|^{*T}_L(\nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)}) - \nabla R_\delta ^{n_{(1)}}(^*))|\)  and \(I_2 = (n\delta )^{1/2}|(\widehat{}_L^{(1)} - _L^{*})\nabla R_\delta ^{n_{(1)}}(\widehat{}_0^{(2)})|\)  are \(o_{}(1)\) , which further implies that \((n\delta )^{1/2}|S_\delta ^{L(1)}(0,\widehat{}^{(2)},\widehat{}^{(1)}_L) - S_\delta ^L(0,^*,^*_L)| = o_{}(1)\) . This shall hold similarly for \(S_\delta ^{L(2)}(0,\widehat{}^{(1)},\widehat{}^{(2)}_L)\) . Invoking Slutsky's theorem completes the proof.

Proofs of additional technical lemmas
Under Assumptions REF  - REF , for any fixed \(\) , we have
\(\begin{aligned}[b]&\Vert \nabla R_\delta () - \nabla R()\Vert _{\infty } \lesssim \delta ^{\ell },\\&\Vert \nabla ^2 R_\delta () - \nabla R()\Vert _{\max }\lesssim \delta ^{\ell -1}.\end{aligned}\) 
[Proof of Lemma ]
We focus on proving the result for \(\Vert \nabla ^2 R_\delta () - \nabla ^2 R()\Vert _{\max }\)  and the proof for the other one is very similar. By definition
\(\begin{aligned}[b]&\Vert \nabla ^2 R_\delta () - \nabla ^2 R()\Vert _{\max }\\=&\max _{j,k} |\sum _y w(y)\int _{} y z_jz_k (\int \frac{-1}{\delta ^2}K^{\prime }(\frac{x - ^T}{\delta }) f(x|y,) dx - f^{\prime }(^T|y,)) f(y,) d|\\=&\max _{j,k} |\sum _y w(y)\int _{} y z_jz_k (\int \frac{-1}{\delta }K^{\prime }(u) f(u\delta + ^T|y,) du - f^{\prime }(^T|y,)) f(y,) d|\\=&\max _{j,k} |\sum _y w(y)\int _{} y z_jz_k \int K(u) (f^{\prime }(u\delta + ^T|y,) - f^{\prime }(^T|y,))du f(y,) d|\\=&\max _{j,k} |\sum _y w(y)\int _{} y z_jz_k \int K(u)\frac{(u\delta )^{\ell -1}}{(\ell -1)!} f^{(\ell )}(\tau u\delta + ^T|y,)du f(y,) d|\\\lesssim &(\delta ^{\ell -1}),\end{aligned}\) 
where the first equality is by definition, the second equality follows from a change of variable, the third equality follows from an integration by parts, the last equality follows from Assumptions REF , REF  and the last inequality follows from Assumptions REF , REF . The proof is complete.
Under Assumptions REF  - REF , for any fixed \(\) , we have with probability greater than \(1-(d^{-1})\)
\(\begin{aligned}[b]&\Vert \nabla R^n_\delta () - \nabla R_\delta ()\Vert _{\infty } \lesssim \sqrt{\frac{\log (d)}{n\delta }},\\&\Vert \nabla ^2 R^n_\delta () - \nabla ^2 R_\delta ()\Vert _{\max }\lesssim \sqrt{\frac{\log (d)}{n\delta ^3}}.\end{aligned}\) 
[Proof of Lemma ]
We focus on proving the result for \(\Vert \nabla ^2 R^n_\delta () - \nabla ^2 R_\delta ()\Vert _{\max }\)  and the proof for the other one is very similar. Denote
\(\begin{aligned}[b]T_{ijk} =& (\nabla ^2 \bar{R}^i_\delta () - \nabla ^2 R_\delta ())_{jk}\\=&-w(y_i)y_i\frac{z_{ij}z_{ik}}{\delta ^2}K^{\prime }(\frac{x_i - ^T_i}{\delta }) - (\nabla ^2 R_\delta ())_{jk}.\end{aligned}\) 
By definition, we know \([T_{ijk}] = 0\) . Meanwhile for \(Var[T_{ijk}]\) , we know
\(\begin{aligned}[b][(\nabla ^2 \bar{R}^i_\delta ())_{jk}^2] =& \sum _y \int w(y)^2 \frac{z_j^2z_k^2}{\delta ^4}K^{\prime 2}(\frac{x - ^T}{\delta })f(x|y,)dx f(y,)d\\=&\sum _y \int w(y)^2\frac{z_j^2z_k^2}{\delta ^3}K^{\prime 2}(u)f(u\delta + ^T|y,)du f(y,)d\\=&(\frac{1}{\delta ^3}),\end{aligned}\) 
and with a similar derivation \((\nabla ^2 R_\delta ())^2_{jk} = (\frac{1}{\delta ^2})\) . This shows that \(Var[T_{ijk}] = (\frac{1}{\delta ^3})\) . Since \(|T_{ijk}|\lesssim \frac{M_n^2}{\delta ^2}\) , by Bernstein inequality we can show that with probability greater than \(1 - (d^{-1})\)
\(\begin{aligned}[b]\Vert \nabla ^2 R^n_\delta () - \nabla ^2 R_\delta ()\Vert _{\max }\lesssim \sqrt{\frac{\log (d)}{n\delta ^3}}.\end{aligned}\) 
This completes the proof.
Under Assumptions REF  - REF , it holds that
\(\begin{aligned}[b]&\Vert \nabla R (\widehat{}) - \nabla R (^*)\Vert _{\infty } \lesssim M_n\Vert \widehat{} - ^*\Vert _1,\\&\Vert \nabla ^2 R (\widehat{}) - \nabla ^2 R (^*)\Vert _{\max } \lesssim M_n\Vert \widehat{} - ^*\Vert _1.\end{aligned}\) 
[Proof of Lemma ]
Here we prove the second inequality and the first one should follow similarly. By definition,
\(\begin{aligned}[b]&\Vert \nabla ^2 R (\widehat{}) - \nabla ^2 R (^*)\Vert _{\max }\\=&\max _{j,k}\bigg |\sum _yw(y)\int _{}z_jz_k \big [f^{\prime }(\widehat{}^T|y,) - f^{\prime }(^{*T}|y,)\big ]f(y,)d\bigg |\\\le &\Vert \widehat{} - ^*\Vert _1M_n|f^{\prime \prime }|_\infty \max _{j,k}\sum _{y}[|Z_jZ_k||Y = y]\\\lesssim &M_n\Vert \widehat{} - ^*\Vert _1.\end{aligned}\) 
Under the conditions in Theorem REF , we have
\((n\delta )^{1/2}\bigg |^{*T}\bigg (\nabla R_\delta ^{n_{(j)}}(\widehat{}^{(k)}) - \nabla R_\delta ^{n_{(j)}}(^*) - \nabla ^2 R_\delta ^{n_{(j)}}(^*)(\widehat{}^{(k)} - ^*)\bigg )\bigg | = o_{}(1),\) 
for \((j,k)\in \lbrace (1,2),(2,1)\rbrace \) .

[Proof of Lemma ]
With some algebra we obtain
\(\begin{aligned}[b]&\bigg |^{*T}\bigg (\nabla R_\delta ^{n_{(j)}}(\widehat{}^{(k)}) - \nabla R_\delta ^{n_{(j)}}(^*) - \nabla ^2 R_\delta ^{n_{(j)}}(^*)(\widehat{}^{(k)} - ^*)\bigg )\bigg |\\=& \bigg |\frac{1}{_j}\sum _{i\in _j}w(y_i)\frac{_i^T^*y_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - \widehat{}^{(k)T}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - \widehat{}^{(k)T}_i}{\delta } - t)dt\bigg |\\\le &\Vert ^*\Vert _1{\frac{1}{_j}\sum _{i\in _j} w(y_i) \frac{_iy_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - \widehat{}^{(k)T}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - \widehat{}^{(k)T}_i}{\delta } - t)dt}_\infty .\end{aligned}\) 
Now we start to analyze \({\frac{1}{_j}\sum _{i\in _j} w(y_i) \frac{_iy_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - \widehat{}^{(k)T}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - \widehat{}^{(k)T}_i}{\delta } - t)dt}_\infty \) . Denote \(G_i= w(y_i) \frac{_iy_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - \widehat{}^{(k)T}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - \widehat{}^{(k)T}_i}{\delta } - t)dt\)  and \(G_{im},1\le m\le d\)  as its coordinates.
Consider the event \(A := \lbrace \Vert \widehat{}^{(k)} - ^*\Vert _{1} \lesssim C\eta _1(n)\rbrace \)  for some constant \(C\) . Notice that for each \(m\)
\(\begin{aligned}[b]&\sum _{y}w(y)\int _{}\frac{z_my}{\delta }\int _x \int _{\frac{x - ^{*T}}{\delta }}^{\frac{x - \widehat{}^{(k)T}}{\delta }}K^{\prime \prime }(t)(\frac{x - \widehat{}^{(k)T}}{\delta } - t)dtf(x|y,)dxf(y,)d\\\stackrel{(u = (x - ^{*T})/\delta )}{=}&\sum _{y}w(y)\int _{}z_my\int _u \int _{u}^{u+\triangle }K^{\prime \prime }(t)(u+\triangle - t)dtf(u\delta +^{*T}|y,)duf(y,)d\\=&\sum _{y}w(y)\int _{}z_my\int _u \int _{u}^{u+\triangle }K^{\prime \prime }(t)(u+\triangle - t)dtf(^{*T}|y,)duf(y,)d\\&~~~~~~~~+ \sum _{y}w(y)\int _{}z_my\int _u u\delta \int _{u}^{u+\triangle }K^{\prime \prime }(t)(u+\triangle - t)dtf^{\prime }(\tau u\delta +^{*T}|y,)duf(y,)d,\end{aligned}\) 
where \(\triangle = \frac{(\widehat{}^{(k)} - )^T}{\delta }\)  and \(\tau \in [0,1]\) . Here the first step is by definition and the last step is from the mean value theorem. Since
\(\begin{aligned}[b]\int _u \int _{u}^{u+\triangle }K^{\prime \prime }(t)(u+\triangle - t)dtdu = 0,\end{aligned}\) 
we can show that the first term on the RHS of the last step is 0. The second term on the RHS can be bounded by \(C^{\prime }\delta |\triangle |^2 \lesssim M_n^2\eta _1(n)^2/\delta \)  for some constant \(C^{\prime }\)  on event \(A\) . This implies that \([G_{im}|A] \lesssim M_n^2\eta _1(n)^2/\delta \) .
Now we look at its variance. For the second moment, for each \(i\in _j,m = 1,\cdots ,d\) , we have
\(\begin{aligned}[b]&\bigg [\bigg (w(y_i)\frac{z_{im}y_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - \widehat{}^{(k)}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - \widehat{}^{(k)}_i}{\delta } - t)dt\bigg )^2\bigg | A\bigg ]\\=&\bigg [\sum _{y}w(y)^2\int _{}\frac{z_m^2}{\delta }\int _u \bigg (\int _{u}^{u+\triangle }K^{\prime \prime }(t)(u+\triangle - t)dt\bigg )^2f(u\delta +^{*T}|y,)duf(y,)d| A\bigg ]\\\le &\bigg [\sum _{y}w(y)^2\int _{}\frac{z_m^2}{\delta }2|f|_\infty |K^{\prime \prime }|^2_\infty \triangle ^4f(y,)d| A\bigg ]\\\lesssim &\frac{M^4_n \eta _1(n)^4}{\delta ^5}.\end{aligned}\) 
Also we know for each \(i\in _j,m = 1,\cdots ,d\) , \(\bigg |\frac{z_{im}y_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i- \widehat{}^{(k)}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - \widehat{}^{(k)T}_i}{\delta } - t)dt\bigg |\lesssim \frac{M_n}{\delta }(\frac{M_n\eta _1(n)}{\delta })^2\)  on \(A\) , and therefore
\(\begin{aligned}[b]Var[G_{im}|A] \lesssim \frac{M^4_n \eta _1(n)^4}{\delta ^5}.\end{aligned}\) 
Therefore, applying Bernstein inequality with \(M_n \sqrt{\frac{\log (d)}{n\delta }} = (1)\) , we can obtain that
\(\begin{aligned}[b]\bigg (\max _m|\frac{1}{_j}\sum _{i\in _j}G_{im} - G_m| > \frac{M_n^2\eta _1(n)^2}{\delta }\sqrt{\frac{\log (d)}{n\delta ^3}}~\big |A \bigg )\le (d^{-1}),\end{aligned}\) 
which further implies that
\(\begin{aligned}[b]&\bigg (\max _m|\frac{1}{_j}\sum _{i\in _j}G_{im} - G_m|> \frac{M_n^2\eta _1(n)^2}{\delta }\sqrt{\frac{\log (d)}{n\delta ^3}}\bigg )\\\le & \bigg (\max _m|\frac{1}{_j}\sum _{i\in _j}G_{im} - G_m| > \frac{M_n^2\eta _1(n)^2}{\delta }\sqrt{\frac{\log (d)}{n\delta ^3}}~\big |A \bigg )+ (A^C) \\= & o(1),\end{aligned}\) 
where the last step follows from Assumption REF .
\(\begin{aligned}[b]{\frac{1}{_j}\sum _{i\in _j} w(Y_i)\frac{_iY_i}{\delta }\int _{\frac{X_i - ^{*T}_i}{\delta }}^{\frac{X_i - \widehat{}^{(k)}_i}{\delta }}K^{\prime \prime }(t)(\frac{X_i - \widehat{}^{(k)}_i}{\delta } - t)dt}_\infty =_{}(\frac{M_n^2\eta _1(n)^2}{\delta }).\end{aligned}\) 
Under the conditions of Theorem REF , the desired result holds. This completes the proof.
Under the conditions of Theorem REF ,
let \(s^{\prime } = \Vert ^*\Vert _1\)  and \(\xi > 0\)  is some constant. If \(s^{\prime }\big (M_n\eta _1(n)\vee \sqrt{\frac{\log (d)}{n\delta ^3}}\vee \delta ^{\ell -1}\big ) = o(1)\) , then with probability tending to one it holds that \(\kappa _D(s^{\prime }) \ge \kappa /\sqrt{2}\) , where
\(\kappa _D(s^{\prime }) = \min \bigg \lbrace \frac{s^{\prime 1/2} (^T \nabla ^2_{} R^n_{\delta }(\widehat{}))^{1/2} }{\Vert _{s^{\prime }}\Vert _1 } : \in ^{d-1} \backslash \lbrace 0\rbrace , \Vert _{s^{\prime c}}\Vert _1 \le \xi \Vert _{s^{\prime }}\Vert _1\bigg \rbrace .\) 
This proof is similar to the proof of Lemma J.3 in [1]} so we only give a sketch here. Firstly we have
\(\begin{aligned}[b]\kappa _D(s^{\prime })^2 \ge \min \bigg \lbrace \frac{^T \nabla ^2_{} R^n_{\delta }(\widehat{})}{\Vert \Vert _2^2 } : \in ^{d-1} \backslash \lbrace 0\rbrace , \Vert _{s^{\prime c}}\Vert _1 \le \xi \Vert _{s^{\prime }}\Vert _1\bigg \rbrace .\end{aligned}\) 
Similar to the proof of Theorem REF , Lemmas , ,  and  together imply that
\(\begin{aligned}[b]&\bigg |\frac{^T (\nabla ^2_{} R^n_{\delta }(\widehat{}) - \nabla ^2_{} R(^*))}{\Vert \Vert _2^2 }\bigg |\\=&\bigg |\frac{^T\big [\frac{1}{2} \big ( \nabla ^2_{} R^{n_{(1)}}_{\delta }(\widehat{}^{(2)}) + \nabla ^2_{} R^{n_{(2)}}_{\delta }(\widehat{}^{(1)}) \big )- \nabla ^2_{} R(^*)\big ]}{\Vert \Vert _2^2 }\bigg |\\\lesssim & s^{\prime }(\xi + 1 )^2 \big [M_n\eta _1(n)\vee \sqrt{\frac{\log (d)}{n\delta ^3}}\vee \delta ^{\ell -1}\big ] = o_{}(1),\end{aligned}\) 
where the inequality is because \(\Vert \Vert _1^2 \le s^{\prime }(\xi + 1 )^2 \Vert \Vert _2^2\) .
Therefore for \(n\)  large enough, we have
\(|\frac{^T (\nabla ^2_{} R^n_{\delta }(\tilde{}) - \nabla ^2_{} R(^*))}{\Vert \Vert _2^2 }| \le \frac{1}{2}\kappa ^2\) . This implies \(\kappa _D(\tilde{s}^*)^2\ge \frac{1}{2}\kappa ^2\)  with probability tending to 1. This completes the proof.
Under the same conditions as in Theorem REF , for \(j = 1,2\) , it holds that
\((n\delta )^{1/2}|^{*T}(\nabla R_\delta ^{n_{(j)}}(0,^*) - \nabla R_\delta ^{n_{(j)}}(^*)) + \theta ^*^{*T}\nabla ^2_{\cdot \theta }R(^*)| = o_{}(1).\) 


By definition
\(\begin{aligned}[b]& |^{*T}(\nabla R_\delta ^{n_{(j)}}(0,^*) - \nabla R_\delta ^{n_{(j)}}(^*)) + \theta ^*^{*T}\nabla ^2_{\cdot \theta }R(^*)| \\&\le \Vert ^*\Vert _1\bigg [\underbrace{\Vert \nabla R_\delta ^{n_{(j)}}(0,^*) - \nabla R_\delta ^{n_{(j)}}(^*) + \theta ^*\nabla ^2_{\cdot \theta }R^{n_{(j)}}_\delta (^*)\Vert _\infty }_{I_1} + \\&~~~~~~~~~\underbrace{\Vert \theta ^*(\nabla ^2_{\cdot \theta }R^{n_{(j)}}_\delta (^*) - \nabla ^2_{\cdot \theta }R_\delta (^*))\Vert _\infty }_{I_2} + \underbrace{\Vert \theta ^*(\nabla ^2_{\cdot \theta }R_\delta (^*) - \nabla ^2_{\cdot \theta }R(^*))\Vert _\infty }_{I_3}\bigg ].\end{aligned}\) 
For \(I_1\) , similar to Lemma , we can write
\(\begin{aligned}[b]&|\nabla R_\delta ^{n_{(j)}}(0,^*) - \nabla R_\delta ^{n_{(j)}}(^*) + \theta ^*\nabla ^2_{\cdot \theta }R^{n_{(j)}}_\delta (^*)|\\=&\bigg |\frac{1}{_j}\sum _{i\in _j}w(y_i)\frac{y_i_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - _0^{*T}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - _0^{*T}_i}{\delta } - t)dt\bigg |\\:=&|\frac{1}{_j}\sum _{i\in _j} G_i|,\end{aligned}\) 
where \(G_i = w(y_i)\frac{y_i_i}{\delta }\int _{\frac{x_i - ^{*T}_i}{\delta }}^{\frac{x_i - _0^{*T}_i}{\delta }}K^{\prime \prime }(t)(\frac{x_i - _0^{*T}_i}{\delta } - t)dt \in ^d\) . Thus, similar to the derivation of Lemma , for each \(1\le m\le d\) , we can show that
\([G_{im}]\lesssim M_n^2\theta ^{*2}/\delta \) , \(|G_{im}|\lesssim M_n^3\theta ^{*2}/\delta ^3\)  and \(Var [G_{im}]\lesssim M_n^4\theta ^{*4}/\delta ^5\) , and thus applying Bernstein inequality yields
\(I_1 = _{}(\frac{M_n^2\theta ^{*2}}{\delta }).\) 
Meanwhile, Lemma  and Lemma  imply that \(I_2 = _{}(\theta ^*\sqrt{\frac{\log (d)}{n\delta ^3}})\)  and \(I_3 = (\theta ^*\delta ^{\ell -1})\) . Combing the above results, we obtain
\(\begin{aligned}[b]|^{*T}(\nabla R_\delta ^{n_{(j)}}(0,^*) - \nabla R_\delta ^{n_{(j)}}(^*)) + \theta ^*^{*T}\nabla ^2_{\cdot \theta }R(^*)| = _{}(\frac{\Vert ^*\Vert _1\theta ^*}{\delta }(M_n^2\theta ^* \vee \sqrt{\frac{\log (d)}{n\delta }} \vee \delta ^\ell )).\end{aligned}\) 
This together with the condition in Theorem REF  completes the proof.
Let \(s^{\prime }= \Vert ^{*}\Vert _0\) . Suppose Assumptions REF -REF  hold, and \(\lambda _{\min }(\nabla ^2_{,}R(^*))\ge c\)  for some constant \(c>0\) .
If we choose \(\delta \asymp (\log (d)/n)^{1/(2\ell +1)}\)  and tuning parameter \(\lambda ^{\prime } \asymp \Vert ^*\Vert _1(M_n\eta _1(n) + (\log (d)/n)^{(\ell -1)/(2\ell +1)})\)  and \(s^{\prime }(M_n\eta _1(n) + (\log (d)/n)^{(\ell -1)/(2\ell +1)}) = o(1)\) , then it holds that
\(\begin{aligned}[b]\Vert \widehat{} - ^*\Vert _1 \lesssim \Vert ^*\Vert _1 s^{\prime }(M_n\eta _1(n) + (\log (d)/n)^{(\ell -1)/(2\ell +1)}).\end{aligned}\) 
Here we focus on the rate for \(\widehat{}^{(1)}\)  and the result will follow accordingly.
Denote \(\widehat{\triangle } = \widehat{}^{(1)} - ^*\) . By definition, we can show that \(\Vert \widehat{\triangle }_{\tilde{s}^{*c}}\Vert _1 \le \Vert \widehat{\triangle }_{\tilde{s}^{*}}\Vert _1\) .
With some algebra, we will get
\(\begin{aligned}[b]\widehat{\triangle }^T\nabla ^2_{} R^{n_{(1)}}_{\delta }(\widehat{}^{(2)})\widehat{\triangle }&= \widehat{\triangle }^T(\nabla ^2_{\cdot }R^{n_{(1)}}_{\delta }(\widehat{}^{(2)}) ^*) - \widehat{\triangle }^T(\nabla ^2_{\cdot } R^{n_{(1)}}_{\delta }(\widehat{}^{(2)}) \widehat{}^{(1)})\\&\le \Vert \widehat{\triangle }\Vert _1\Vert \nabla ^2_{\cdot }R^{n_{(1)}}_{\delta }(\widehat{}^{(2)})^*\Vert _\infty + \Vert \widehat{\triangle }\Vert _1\Vert \nabla ^2_{\theta } R^{n_{(1)}}_{\delta }(\widehat{}^{(2)}) - \nabla ^2_{} R^{n_{(1)}}_{\delta }(\widehat{}^{(2)})\widehat{}^{(1)} \Vert _\infty .\end{aligned}\) 
By definition, the second term is bounded by \(\lambda ^{\prime }\Vert \widehat{\triangle }\Vert _1\) . For the first term, recall by definition \(\nabla ^2_{\cdot }R(^*)^* = 0\) , and thus similar to the proof of Theorem REF , Lemma ,  and  imply that with probability approaching to 1
\(\Vert \nabla ^2_{\cdot }R^{n_{(1)}}_{\delta }(\widehat{}^{(2)})^*\Vert _\infty \lesssim \Vert ^*\Vert _1\bigg [M_n\eta _1(n) \vee \sqrt{\frac{\log (d)}{n\delta ^3}}\vee \delta ^{\ell -1}\bigg ],\) 
and thus with the choice of \(\lambda ^{\prime }\)  it holds that \(\widehat{\triangle }^T\nabla ^2_{} R^{n_{(1)}}_{\delta }(\widehat{}^{(2)})\widehat{\triangle } \lesssim \lambda ^{\prime }\Vert \widehat{\triangle }\Vert _1\) . This together with Lemma  implies that \(\Vert \widehat{\triangle }\Vert _1\lesssim s^{\prime }\lambda ^{\prime }\)  with high probability. This completes the proof.
Under the same conditions of Theorem REF , if \(U\)  is a proper kernel of order \(\ell \)  satisfying the same condition as \(K\)  in Assumption REF , and in addition \(U\)  is \(\ell \)  times continuously differentiable and \(U^{(i)}\)  degenerates at the boundary for \(i = 0,\cdots ,\ell -1\) , then when \(\sqrt{\frac{\log (d)}{nh^{2\ell +1}}} + (M_n\eta _1(n) \vee h)^\zeta = o(1)\) , it holds that
\(|\widehat{\mu } - ^{*T}^*| \lesssim \Vert ^*\Vert _1\bigg (\eta _2(n) + \sqrt{\frac{\log (d)}{nh^{2\ell +1}}} + (M_n\eta _1(n) \vee h)^\zeta \bigg ).\) 


It suffices to proof the results for \(\widehat{}^{(1)T}\widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})\) . By definition,
\(\begin{aligned}[b]|^{*T}T^{(\ell )}(^*) - \widehat{}^{(1)T}\widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})|\le |(^* - \widehat{})^T \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)}) | + |^{*T}(T^{(\ell )}(^*) - \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)}))|.\end{aligned}\) 
We firstly look at \(\Vert T^{(\ell )}(^*) - \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})\Vert _\infty \) . Direct calculation gives that
\(\begin{aligned}[b]&\Vert T^{(\ell )}(^*) - \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})\Vert _\infty \\\le &\Vert T^{(\ell )}(^*) - T^{(\ell )}(\widehat{}^{(2)})\Vert _\infty +\Vert T^{(\ell )}(\widehat{}^{(2)}) - \tilde{T}_{h,U}^{(\ell )}(\widehat{}^{(2)})\Vert _\infty +\Vert \tilde{T}_{h,U}^{(\ell )}(\widehat{}^{(2)}) - \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})\Vert _\infty ,\end{aligned}\) 
where \(\tilde{T}_{h,U}^{(\ell )}(\widehat{}^{(2)}) := \sum _{y \in \lbrace -1,1\rbrace }w(y)y\int \frac{ }{h^{1+\ell }}\int U^{(\ell )}\bigg (\frac{\widehat{}^{(2)T}- x}{h}\bigg )f(x|y,)dx f(y,)d\) . For the first term, we have
\(\begin{aligned}[b]&\Vert T^{(\ell )}(^*) - T^{(\ell )}(\widehat{}^{(2)})\Vert _\infty \\=&{\sum _{y \in \lbrace -1,1\rbrace }w(y)\int y(f^{(\ell )}(^{*T}|y,) - f^{(\ell )}(\widehat{}^{(2)T}|y,))f(y,) d}_\infty \\=&_{}((M_n\eta _1(n))^{\zeta } ),\end{aligned}\) 
where the last step follows from Assumption REF . For the second term, notice that we can rewrite
\(\begin{aligned}[b]&\tilde{T}_{h,U}^{(\ell )}(\widehat{}^{(2)})\\=&\sum _{y \in \lbrace -1,1\rbrace }w(y)y\int \frac{ }{h}\int U\bigg (\frac{ \widehat{}^{(2)T}- x}{h}\bigg )f^{(\ell )}(x|y,)dx f(y,)d\\=&\sum _{y \in \lbrace -1,1\rbrace }w(y)y\int \int U(u)f^{(\ell )}(\widehat{}^{(2)T}- hu|y,)du f(y,)d,\end{aligned}\) 
where the first step is by repeated integration by parts and the second step is by a change of variable. This implies that
\(\begin{aligned}[b]&\Vert T^{(\ell )}(\widehat{}^{(2)}) - \tilde{T}_{h,U}^{(\ell )}(\widehat{}^{(2)})\Vert _\infty \\=&{\sum _{y \in \lbrace -1,1\rbrace }w(y)y\int \int U(u)\bigg (f^{(\ell )}(\widehat{}^{(2)T}|y,) -f^{(\ell )}(\widehat{}^{(2)T}-hu|y,)\bigg )du f(y,)d}_\infty \\=&(h^{\zeta }).\end{aligned}\) 
Finally, similar to the proof of Theorem REF , we can show that the third term on the RHS of (REF ) is \(_{}(\sqrt{\frac{\log (d)}{nh^{2\ell + 1}}})\)  based on Lemmas  and . Putting all three terms together we have
\(\begin{aligned}[b]\Vert T^{(\ell )}(^*) - \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})\Vert _\infty = _{}\bigg (\sqrt{\frac{\log (d)}{nh^{2\ell +1}}} + (M_n\eta _1(n) \vee h)^\zeta \bigg ),\end{aligned}\) 
and thus \(\Vert \widehat{T}_{h,U}^{(\ell ),n_{(1)}}(\widehat{}^{(2)})\Vert  = _{}(1)\)  by the condition of this lemma. Plugging this back to (REF ) gives the desired result. This completes the proof.
Under the same conditions of Theorem REF , if \(L\)  is a proper kernel of order \(\ell \)  satisfying Assumption REF  and \(g^{\ell } + \sqrt{\frac{\log (d)}{ng}} + M_n\eta _1(n) = o(1)\) , then we have
\(\begin{aligned}[b]|\widehat{\sigma }^2 - ^{*T}^*^* | \lesssim \Vert ^*\Vert _1^2\bigg (\eta _2(n) + g^{\ell } + \sqrt{\frac{\log (d)}{ng}} + M_n\eta _1(n)\bigg ).\end{aligned}\) 
It suffices to show the convergence rate of \(\tilde{\mu }_K\widehat{}^{(1)T}\widehat{H}_{g,K}^{n_{(1)}}(\widehat{}^{(2)})\widehat{}^{(1)}\) .
Similar to the proof of Theorem REF , following Lemmas , ,  and , we can show that
\(\begin{aligned}[b]\Vert \widehat{H}_{g,K}^{(1)}(\widehat{}^{(2)}) - H(^*)\Vert _{\max } = _{}\bigg (g^{\ell } + \sqrt{\frac{\log (d)}{ng}} + M_n\eta _1(n)\bigg ).\end{aligned}\) 
Since \(\Vert H(^*)\Vert _{\max } =(1)\) , we know \(\Vert \widehat{H}_{g,K}^{(1)}(\widehat{}^{(2)})\Vert _{\max } = _{}(1)\)  when \(g^{\ell } + \sqrt{\frac{\log (d)}{ng}} + M_n\eta _1(n) = o(1)\) .
Now by triangle inequality, we obtain
\(\begin{aligned}[b]&|\widehat{}^{(1)T}\widehat{H}_{g,K}^{(1)}(\widehat{}^{(2)})\widehat{}^{(1)} - ^{*T}H(^*)^*|\\ \le & \Vert \widehat{}^{(1)} - ^*\Vert ^2_1\Vert \widehat{H}_{g,K}^{(1)}(\widehat{}^{(2)})\Vert _{\max } + 2\Vert ^{*T}\widehat{H}_{g,K}^{(1)}(\widehat{}^{(2)})\Vert _\infty \Vert \widehat{}^{(1)} - ^*\Vert _1 + |^{*T}(\widehat{H}_{g,K}^{(1)}(\widehat{}^{(2)}) - H(^*))^*|.\end{aligned}\) 
This implies that
\(\begin{aligned}[b]|\widehat{\sigma }^2 - ^{*T}^*^* | = \Vert ^*\Vert _1^2_{}\bigg (\eta _2(n) + g^{\ell } + \sqrt{\frac{\log (d)}{ng}} + M_n\eta _1(n)\bigg ).\end{aligned}\) 
This completes the proof.
[Lemma 6.1 of [2]}]
Let \(\lbrace _m\rbrace \)  and \(\lbrace _m\rbrace \)  be sequences of random vectors. If \(\Vert _m\Vert  = _{}(A_m)\)  conditional on \(\lbrace _m\rbrace \)  for a sequence of positive constants \(\lbrace A_m\rbrace \) , then \(\Vert _m\Vert  = _{}(A_m)\)  unconditionally.

Practical Considerations
In practice, in order to apply the smoothed decorrelated score test presented in Section REF , we need to select the bandwidth and regularization parameter in the initial estimators \(\hat{}\)  and \(\hat{}\) , two bandwidths for the plug-in asymptotic bias and variance estimators in (REF ) and (REF ), and the final bandwidth \(\delta \)  for the score test statistic. In this section, we discuss practical considerations for the proposed test as well as the data-driven selection procedure for \(\delta \) .
We use a path-following algorithm to compute the initial estimator \(\hat{}\)  and apply a two way cross-validation approach to choose the tuning parameters \((\delta ,\lambda )\)  in the optimization (REF ); see [3]} for details. The initial values of the path-following algorithm used in the simulation are shown in the appendix Section .
To compute the Dantzig estimator \(\hat{}\) , we use the “flare" package in R. The same cross-validation method can be used to select the tuning parameters \((\delta ,\lambda ^{\prime })\)  in (REF ). Meanwhile, we find the empirical performance of the Dantzig estimator \(\hat{}\)  is not very sensitive to the choice of \((\delta ,\lambda ^{\prime })\) . In the simulation and real data analysis, to ease the computation, we set \(\lambda ^{\prime } = 2(\log d/n)^{1/5}\)  and \(\delta = 1\)  in (REF ). Notice that in rare cases, the Dantzig solver may become unstable numerically when the Hessian \(\nabla ^2 R^n_\delta (\widehat{})\)  is ill-conditioned. In this case, we will firstly project it onto the cone of positive definite matrices and then plug-in the projected matrix into (REF ) to solve \(\widehat{}\) .
For the bandwidth parameters \(h\)  and \(g\)  in the plug-in bias and variance estimators, Lemma  and Lemma  imply that the theoretical optimal order for \(h\)  and \(g\)  are \((\log (d)/n)^{2\ell + 2\zeta + 1}\)  (if \(M_n\eta _1(n) = o(h)\)  which holds under mild conditions) and \((\log (d)/n)^{2\ell + 1}\) , respectively. In practice, we recommend choosing \(h = c_1(\log (d)/n)^{2\ell ^{\prime } + 3}\)  and \(g = c_2(\log (d)/n)^{2\ell ^{\prime } + 1}\)  for some \(c_1,c_2 > 0\)  (e.g., \(c_1=c_2=2\)  in our simulation), where \(\ell ^{\prime }\)  is the order of the kernel function \(K\) .
Finally, for the main bandwidth parameter \(\delta \)  appearing in the score test statistic, if a pre-specified value is preferred, we recommend choosing \(\delta = c n^{-1/(2\ell ^{\prime } + 1)}\)  for some constant \(c >0\)  (e.g., \(c=1\) ). When using
our proposed data-driven bandwidth selector, notice that another bandwidth parameter \(b\) , whose optimal order is \((\log (d)/n)^{1/(2\ell + 2r + 1)}\)  (see the discussion after Theorem ), is required for estimating the squared bias \(SB(\delta )\) . Similarly, we suggest choosing \(b=c_0(\log (d)/n)^{1/(2\ell ^{\prime } + 2r^{\prime } + 1)}\)  for some \(c_0\)  (e.g., \(c_0=0.5\) ), where \(r^{\prime }\)  is the order of kernel \(J\) .
To further automate and robustify the whole procedure with data-driven bandwidth \(\widehat{\delta }\) , we provide an alternative variance estimator for the score function which does not require an additional bandwidth parameter. Given \(\widehat{\delta }\) , the numerator of the score statistic \(\widehat{U}_n\)  can be written as
\(\begin{aligned}[b]\widehat{S}_{\widehat{\delta }}(0,\widehat{}) - \widehat{\delta }^{\ell }\widehat{\mu }=&\widehat{}^T\nabla R_{\widehat{\delta }}^n(\widehat{}_0) - \widehat{\delta }^{\ell }\gamma _{K,\ell }\widehat{}^T\widehat{T}_{h,U}^{\ell }(\widehat{})=D_{\hat{\delta },h}^n(\hat{},\hat{}),\end{aligned}\) 
where
\(D_{\delta ,h}^n(,) = \frac{1}{n} \sum _{i=1}^n w(y_i)y_i \frac{^T_i}{\delta }M(x_i,_i;),\) 
and \(M(x_i,_i;) = K\big (\frac{ {}_0^{T}_i - x_i}{h} \big )- \frac{\delta ^{\ell +1}}{h^{\ell +1}}U^{(\ell )}\big (\frac{ {}^{T}_i - x_i}{h}\big )\) . To this end, we may alternatively seek to estimate the variance
\(\begin{aligned}[b][D_{\delta ,h}^n(,)] = \frac{1}{n} \bigg \lbrace \Big [(w(Y)Y\frac{^T}{\delta }M(X,;))^2\Big ] - \Big [w(Y)Y\frac{^T}{\delta }M(X,;)\Big ]^2\bigg \rbrace \end{aligned}\) 
by replacing the expectations above with sample averages under \(\widehat{\delta },\widehat{}\)  and \(\widehat{}\) . We can see that this variance estimator does not require additional bandwidth parameters as in the pilot estimator (REF ) and thus makes the procedure more convenient in practice. In addition, we find this estimator empirically robust when the bandwidth parameter \(\delta \)  is selected adaptively using the data.

Additional Numerical Results
Computational details of the path-following algorithm
In Section , we apply the path-following algorithm proposed in [3]} to estimate the true coefficient \(\) . We fix the number of stages \(N = 25\) , \(\nu = 0.25\) , \(\eta = 0.25\)  ,\(\epsilon _{tgt} = 0.0001\)  and set \(\Omega = \lbrace : \Vert \Vert _2 \le 10^3\rbrace \)  for each parameter in Algorithm 2.1 and 2.2 in [3]}.

Normal Q-Q plots in Section  REF
Figure REF  contains the Normal Q-Q plots for the test statistics obtained from the experiments in Section REF  under both scenarios. We can see that the Q-Q plots from the proposed SDS method appear close to Gaussian, while the Q-Q plots for DS and Honest method deviate from standard Gaussian.

Empirical power in Section  REF
Figure REF  shows the empirical power under the Heteroskedastic Uniform scenario in Section REF . Similar to the figures for the Heteroskedastic Gaussian scenario, the empirical power converges to 1 as the magnitude of \(\beta ^*\)  becomes larger. In addition, as the correlation between \(\)  becomes larger, the power also decreases.

Bandwidth selection in Section  REF
Finally, we investigate whether the data-driven bandwidth is close to the theoretically optimal bandwidth that minimizes the MSE. We focus on the Heteroskedastic Gaussian scenario and consider the setting with \(n = 800, d = 50, s = 10, \rho = 0.2\)  and set \(\beta ^*_1 = \cdots = \beta ^*_{10} = 1/\sqrt{10}\) . We compare \(M(\delta )\)  with \(\widehat{M}(\delta )\) , where \(\widehat{M}(\delta )\)  is obtained using the implementation discussed in Section , but with different bandwidth \(b = 0.15,0.2,0.25,0.3\)  for squared bias estimation. Notice that since directly obtaining \(M(\delta )\)  is hard, we apply Monte Carlo integration to approximate this quantity.
Figure REF  shows the true \(M(\delta )\)  as well as \(\widehat{M}(\delta )\)  in each case. The red line is the true MSE \(M(\delta )\)  approximated by Monte Carlo method, and the black solid line is the average estimated MSE \(\widehat{M}(\delta )\)  with different bandwidths \(b\)  over 100 repetitions. The black dashed line is the one standard deviation band around \(\widehat{M}(\delta )\)  over these repetitions.
As we can see, for all cases, \(M(\delta )\)  lies within the one standard deviation band around \(\widehat{M}(\delta )\)  and the minimizer of \(M(\delta )\) 
is very close to the minimizer of the average \(\widehat{M}(\delta )\) . This experiment confirms that the data-driven bandwidth \(\widehat{\delta }\)  is close to the optimal \(\delta ^*\) , provided that the bandwidth \(b\)  for squared-bias estimation lies in a suitable range.

Additional results from ChAMP analysis
Table REF  lists the five most significant variables using the DS and Honest approach. Notice that the significant variables identified by the proposed approach are also in the list for the DS approach. Honest yields very different set of variables compared to the other two.
<FIGURE><FIGURE><FIGURE><TABLE>