During training, a certain percentage of words are masked for use in prediction. If both "San" and "Francisco" were masked, BERT would not be able to use information when decoding one of the words to help in decoding the other. masking the input introduces a few disadvantages mentioned in <cite>[11]</cite> .