A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension
Identifying the relevant variables for a classification model with correct
confidence levels is a central but difficult task in high-dimension. Despite
the core role of sparse logistic regression in statistics and machine learning,
it still lacks a good solution for accurate inference in the regime where the
number of features $p$ is as large as or larger than the number of samples $n$.
Here, we tackle this problem by improving the Conditional Randomization Test
(CRT). The original CRT algorithm shows promise as a way to output p-values
while making few assumptions on the distribution of the test statistics. As it
comes with a prohibitive computational cost even in mildly high-dimensional
problems, faster solutions based on distillation have been proposed. Yet, they
rely on unrealistic hypotheses and result in low-power solutions. To improve
this, we propose \emph{CRT-logit}, an algorithm that combines a
variable-distillation step and a decorrelation step that takes into account the
geometry of $\ell_1$-penalized logistic regression problem. We provide a
theoretical analysis of this procedure, and demonstrate its effectiveness on
simulations, along with experiments on large-scale brain-imaging and genomics
datasets.