@inproceedings{
zhao2024metric,
title={Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation},
author={Yizhou Zhao and Hengwei Bian and Kaihua Chen and Pengliang Ji and Liao Qu and Shao-yu Lin and Weichen Yu and Haoran Li and Hao Chen and Jun Shen and Bhiksha Raj and Min Xu},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=GA8TVtxudf}
}
Monocular depth estimation (MDE) is fundamental for deriving 3D scene structures from 2D images. While state-of-the-art monocular relative depth estimation (MRDE) excels in estimating relative depths for in-the-wild images, current monocular metric depth estimation (MMDE) approaches still face challenges in handling unseen scenes. Since MMDE can be viewed as the composition of MRDE and metric scale recovery, we attribute this difficulty to scene dependency, where MMDE models rely on scenes observed during supervised training for predicting scene scales during inference. To address this issue, we propose to use humans as landmarks for distilling scene-independent metric scale priors from generative painting models. Our approach, Metric from Human (MfH), bridges from generalizable MRDE to zero-shot MMDE in a generate-and-estimate manner. Specifically, MfH generates humans on the input image with generative painting and estimates human dimensions with an off-the-shelf human mesh recovery (HMR) model. Based on MRDE predictions, it propagates the metric information from painted humans to the contexts, resulting in metric depth estimations for the original input. Through this annotation-free test-time adaptation, MfH achieves superior zero-shot performance in MMDE, demonstrating its strong generalization ability.
Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation
Yizhou Zhao Hengwei Bian Kaihua Chen Pengliang Ji Liao Qu Shao-yu Lin Weichen Yu Haoran Li Hao Chen Jun Shen Bhiksha Raj Min Xu