@inproceedings{sun-etal-2024-lightvlp,
    title = "{L}ight{VLP}: A Lightweight Vision-Language Pre-training via Gated Interactive Masked {A}uto{E}ncoders",
    author = "Sun, Xingwu  and
      Yang, Zhen  and
      Xie, Ruobing  and
      Lian, Fengzong  and
      Kang, Zhanhui  and
      Xu, Chengzhong",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.918",
    pages = "10499--10510",
    abstract = "This paper studies vision-language (V{\&}L) pre-training for deep cross-modal representations. Recently, pre-trained V{\&}L models have shown great success in V{\&}L tasks. However, most existing models apply multi-modal encoders to encode the image and text, at the cost of high training complexity because of the input sequence length. In addition, they suffer from noisy training corpora caused by V{\&}L mismatching. In this work, we propose a lightweight vision-language pre-training (LightVLP) for efficient and effective V{\&}L pre-training. First, we design a new V{\&}L framework with two autoencoders. Each autoencoder involves an encoder, which only takes in unmasked tokens (removes masked ones), as well as a lightweight decoder that reconstructs the masked tokens. Besides, we mask and remove large portions of input tokens to accelerate the training. Moreover, we propose a gated interaction mechanism to cope with noise in aligned image-text pairs. As for a matched image-text pair, the model tends to apply cross-modal representations for reconstructions. By contrast, for an unmatched pair, the model conducts reconstructions mainly using uni-modal representations. Benefiting from the above-mentioned designs, our base model shows competitive results compared to ALBEF while saving 44{\%} FLOPs. Further, we compare our large model with ALBEF under the setting of similar FLOPs on six datasets and show the superiority of LightVLP. In particular, our model achieves 2.2{\%} R@1 gains on COCO Text Retrieval and 1.1{\%} on refCOCO+.",
}

This paper studies vision-language (V{\&}L) pre-training for deep cross-modal representations. Recently, pre-trained V{\&}L models have shown great success in V{\&}L tasks. However, most existing models apply multi-modal encoders to encode the image and text, at the cost of high training complexity because of the input sequence length. In addition, they suffer from noisy training corpora caused by V{\&}L mismatching. In this work, we propose a lightweight vision-language pre-training (LightVLP) for efficient and effective V{\&}L pre-training. First, we design a new V{\&}L framework with two autoencoders. Each autoencoder involves an encoder, which only takes in unmasked tokens (removes masked ones), as well as a lightweight decoder that reconstructs the masked tokens. Besides, we mask and remove large portions of input tokens to accelerate the training. Moreover, we propose a gated interaction mechanism to cope with noise in aligned image-text pairs. As for a matched image-text pair, the model tends to apply cross-modal representations for reconstructions. By contrast, for an unmatched pair, the model conducts reconstructions mainly using uni-modal representations. Benefiting from the above-mentioned designs, our base model shows competitive results compared to ALBEF while saving 44{\%} FLOPs. Further, we compare our large model with ALBEF under the setting of similar FLOPs on six datasets and show the superiority of LightVLP. In particular, our model achieves 2.2{\%} R@1 gains on COCO Text Retrieval and 1.1{\%} on refCOCO+.
Sun, Xingwu Yang, Zhen Xie, Ruobing Lian, Fengzong Kang, Zhanhui Xu, Chengzhong
LightVLP: A Lightweight Vision-Language Pre-training via Gated Interactive Masked AutoEncoders