Self-supervised Learning (SSL) has been popular in recent years without the expensive data annotation. By designing pretext tasks, SSL can learn semantic representations and generalizes well in downstream tasks. [1]} first proposes the pretext of predicting the relative location of image patches. Methods in [2]}, [3]}, [4]} design the pretext task of rotation prediction that has shown promising results in representative feature learning. In [5]}, a jigsaw puzzle prediction task is proposed and generalizes well in domain adaptation object recognition. Although achieving remarkable progress, the above methods rely on the pretext task of data augmentation. Recently, MAE [6]} first masks random patches of the input image and reconstructs the missing pixels with the simple autoencoder framework, showing promising results in the field of self-supervised learning. ConvMAE [7]} adopts the masked convolution to prevent information leakage and learns more discriminative representations with multi-scale hybrid convolution-transformer than MAE. LoMaR [8]} decreases the computational cost of MAE with a small window
of 7\(\times \) 7 patches on a simple Transformer encode. VideoMAE [9]} and method in [10]} extend the MAE into spatiotemporal representation learning from videos that are more information redundant. In this paper, we explore the effectiveness of masked autoencoders in pre-training 3D object detectors from large-scale point clouds, as they are also information redundant.
