Random mask strategy. Inspired of the MLM strategy for natural language pre-training, we apply the random mask strategy to self-supervised learning. Given a dataset \(Q\)  without manual annotations, and \(\bf e\)  = \((e_1,...,e_n)\)  denote a image of \(n\)  tokens, where \(i\)  = \(1,...,n\) . Let \(\bf m\)  = \((m_1,...,m_n)\)  denote a binary vector of length \(n\) , where \(m_i \in \lbrace 0,1\rbrace \) , representing the mask over image. According to BERT [1]}, the \(\bf m\)  can be obtained with probability \(p\)  by Eq (REF ), and the \(p\)  is 0.15 by default.
\(m_i ={\left\lbrace \begin{array}{ll}1, & prob_i < p \; \\0, & \textup {otherwise}\end{array}\right.}\) 
