Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks
For unsupervised pretraining, mask-reconstruction pretraining (MRP)
approaches randomly mask input patches and then reconstruct pixels or semantic
features of these masked patches via an auto-encoder. Then for a downstream
task, supervised fine-tuning the pretrained encoder remarkably surpasses the
conventional supervised learning (SL) trained from scratch. However, it is
still unclear 1) how MRP performs semantic learning in the pretraining phase
and 2) why it helps in downstream tasks. To solve these problems, we
theoretically show that on an auto-encoder of a two/one-layered convolution
encoder/decoder, MRP can capture all discriminative semantics in the
pretraining dataset, and accordingly show its provable improvement over SL on
the classification downstream task. Specifically, we assume that pretraining
dataset contains multi-view samples of ratio $1-\mu$ and single-view samples of
ratio $\mu$, where multi/single-view samples has multiple/single discriminative
semantics. Then for pretraining, we prove that 1) the convolution kernels of
the MRP encoder captures all discriminative semantics in the pretraining data;
and 2) a convolution kernel captures at most one semantic. Accordingly, in the
downstream supervised fine-tuning, most semantics would be captured and
different semantics would not be fused together. This helps the downstream
fine-tuned network to easily establish the relation between kernels and
semantic class labels. In this way, the fine-tuned encoder in MRP provably
achieves zero test error with high probability for both multi-view and
single-view test data. In contrast, as proved by~[3], conventional SL can only
obtain a test accuracy between around $0.5\mu$ for single-view test data. These
results together explain the benefits of MRP in downstream tasks. Experimental
results testify to multi-view data assumptions and our theoretical
implications.