Document Pre-training is an effective strategy to improve document-related downstream tasks [1]}, [2]}, [3]}. [1]} propose LayoutLM that models interaction between texts and layouts in scanned documents using masked language modeling and image-text matching. Later, LayoutLMv2 [2]} is introduced to emphasize the importance of image features for document understanding and present a spatial-aware self-attention mechanism to improve learning relative positional relationship among different text blocks. Most recently, [3]} propose DocFormer that encourages the interaction between image and text modalities by adding an image reconstruction task. The existing pre-training methods perform well when applied to downstream tasks such as document classification and token sequence labeling. However, all of the above methods include the absolute 1-D positional embedding (the so called reading order [1]}) of the tokens in the inputs. Although, this 1-D embedding is a helpful prior knowledge for a holistic understanding of a document, it hinders a model from learning rich geometric relationship among tokens, thus is not beneficial to our value retrieval task, where local geometric relationship between words is essential for prediction. Our method uses permutation-invariant positional encoding to improve model performance.
<FIGURE>