@inproceedings{
huang2024eventguided,
title={Event-Guided Rolling Shutter Correction with Time-Aware Cross-Attentions},
author={Hefei Huang and Xu Jia and Xinyu Zhang and Shengming Li and Huchuan Lu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=Qp0d7CfKq8}
}
Many consumer cameras with rolling shutter (RS) CMOS would suffer undesired distortion and artifacts, particularly when objects experiences fast motion. The neuromorphic event camera, with high temporal resolution events, could bring much benefit to the RS correction process. In this work, we explore the characteristics of RS images and event data for the design of the rolling shutter correction (RSC) model. Specifically, the relationship between RS images and event data is modeled by incorporating time encoding to the computation of cross-attention in transformer encoder to achieve time-aware multi-modal information fusion. 
Features from RS images enhanced by event data are adopted as keys and values in transformer decoder, providing source for appearance, while features from event data enhanced by RS images are adopted as queries, providing spatial transition information. 
By embedding the time information of the desired GS image into the query, the transformer with deformable attention is capable of producing the target GS image.
To enhance the model's generalization ability, we propose to further self-supervise the model by cycling between time coordinate systems corresponding to RS images and GS images.
Extensive evaluations over both synthetic and real datasets demonstrate that the proposed method performs favorably against state-of-the-art approaches.
Event-Guided Rolling Shutter Correction with Time-Aware Cross-Attentions
Hefei Huang Xu Jia Xinyu Zhang Shengming Li Huchuan Lu