NRSfM is known to be an under-constrained problem if the shape is allowed to deform arbitrarily in each observation. To make this problem tractable, a standard assumption is that in each frame the 3D shape is a linear combination of a small number of basis shapes [1]}. With this assumption, NRSfM is formulated as factorizing the stacked observation matrix into three component matrices: cameras, coefficients and basis. Previous researches exploit various constraints to solve this factorization problem, involving orthogonal constraint on the camera matrix [2]}, [3]}, restricting the basis to 3D shapes [4]}. Different from those constraints on cameras or basis, another important category of approaches applied constraints to the coefficient matrix, including smooth trajectories over time in original coefficient matrix [5]}, [6]} or in low-dimensional manifold [7]}, prior distributions [8]}, [9]} and spatial smoothness [10]}. In neural-network based models, the latent representation can be thought of as the `coefficients', and Sidhu et al. [11]} first apply latent space constraints for sequential dense reconstruction. These constraints reduce the indeterminacy of the NRSfM task and potentially lead to better reconstruction.
