@inproceedings{
xu2024se,
title={\${SE}(3)\$ Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation},
author={Yinshuang Xu and Dian Chen and Katherine Liu and Sergey Zakharov and Rares Andrei Ambrus and Kostas Daniilidis and Vitor Campagnolo Guizilini},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=yRuJqoWoCs}
}
Incorporating inductive bias by embedding geometric entities (such as rays) as input has proven successful in multi-view learning. However, the methods adopting this technique typically lack equivariance, which is crucial for effective 3D learning. Equivariance serves as a valuable inductive prior, aiding in the generation of robust multi-view features for 3D scene understanding. In this paper, we explore the application of equivariant multi-view learning to depth estimation, not only recognizing its significance for computer vision and robotics but also addressing the limitations of previous research. Most prior studies have either overlooked equivariance in this setting or achieved only approximate equivariance through data augmentation, which often leads to inconsistencies across different reference frames. To address this issue, we propose to embed $SE(3)$ equivariance into the Perceiver IO architecture. We employ Spherical Harmonics for positional encoding to ensure 3D rotation equivariance, and develop a specialized equivariant encoder and decoder within the Perceiver IO architecture. To validate our model, we applied it to the task of stereo depth estimation, achieving state of the art results on real-world datasets without explicit geometric constraints or extensive data augmentation.
SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation
Yinshuang Xu Dian Chen Katherine Liu Sergey Zakharov Rares Andrei Ambrus Kostas Daniilidis Vitor Campagnolo Guizilini