We suppose that the rotation is arbitrary in each frame. A recent work [1]} shows that one needs 6D representation as well as a good mapping to avoid discontinuity while using the polar decomposition for enforcing orthogonality. The rotation estimation network is designed to output a 6D vector. The network is constructed with multiple linear layers, as shown in Fig. REF . The output of the linear layers is reshaped into \(\mathrm {\tilde{M}}_i \in \mathbb {R}^{2 \times 3}\) .
