@inproceedings{
zhong2024vlreader,
title={{VL}-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizer},
author={Humen Zhong and Zhibo Yang and Zhaohai Li and Peng Wang and Jun Tang and Wenqing Cheng and Cong Yao},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=eGNXda1ooK}
}
Text recognition is an inherent integration of vision and language,
encompassing the visual texture in stroke patterns and the semantic
context among the character sequences. Towards advanced text
recognition, there are three key challenges: (1) an encoder capable
of representing the visual and semantic distribution; (2) a decoder
supervises the alignment between vision and semantics; and (3)
consistency in the framework during pre-training and fine-tuning.
Inspired by masked autoencoding, a successful pre-training strategy
in both vision and language, we propose an innovative scene text
recognition approach, named VL-Reader. The novelty of VL-Reader
lies in that the interplay between vision and language is pervasive
throughout the entire process, not only in the encoding stage but
also the decoding stage, which has been previously overlooked.
Concretely, we first introduce a Masked Visual-Linguistic Recon-
struction (MVLR) objective, which aims at simultaneously modeling
visual and linguistic information. Then, we design a Masked Visual-
Linguistic Decoder (MVLD) to further leverage bi-modal feature
interaction. The architecture of VL-Reader maintains consistency
from training to inference. In the pre-training stage, VL-Reader
reconstructs both masked visual and text tokens, while in the fine-
tuning stage, the network degrades to reconstruct all characters
from an image without any masked regions. VL reader achieves an
average accuracy of 97.1% on six typical datasets, surpassing the
SOTA by 1.1%. The improvement was even more significant on chal-
lenging datasets. The results demonstrate that vision and language
reconstructor can serve as an effective scene text recognizer.
VL-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizer
Humen Zhong Zhibo Yang Zhaohai Li Peng Wang Jun Tang Wenqing Cheng Cong Yao