The inputs to our pre-trained model are the OCR words \(\lbrace w_{1},w_{2},...,w_{N}\rbrace \)  associated with their locations \(\lbrace b_{1},b_{2},...,b_{N}\rbrace \) . The word/location embedding protocol and the transformer structure of our pre-trained model are the same as those of our value retrieval model such that the pre-trained model can be directly used for initializing the parameters of the value retrieval model. Specifically, the word embedding, \(f^{txt}_{w_{j}}\) , is constructed by using a simple look up table. Previous works [1]}, [2]}, [3]} require the input text sorted in the reading order, so that they can process texts of the document in a similar way of processing languages. They leverage this prior knowledge by adding the ranking of each word as the 1-D positional embedding in the final location embedding, \(f^{loc}_{b_{j}}\) . This 1-D embedding provides a holistic view of the geometric relationship of words. However, it introduces extra dependence on the OCR engines (most SOTA OCR engines do not have the capability of sorting detected OCR words in the reading order), and it also restricts the model from learning local geometric relations between words in a flexible way. To encourage a model to better learn the local geometric relations, we exclude the 1-D positional embedding and only encode the 2-D bounding-box location (top-left, bottom-right, width and height) of each word using a lookup table. For simplicity, we only use the masked language modeling as the pre-training objective. We show in Section  that our method improves the SOTA largely by using this simple pre-training strategy.
