@inproceedings{
zhao2024crossmodal,
title={Crossmodal Few-shot 3D Point Cloud Semantic Segmentation via View Synthesis},
author={Ziyu Zhao and Pingping Cai and Canyu Zhang and Xiaoguang Li and Song Wang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=8KRzIOM7Ar}
}
Cross-modal 2D–3D point cloud semantic segmentation on few-shot-based learning provides a practical approach for borrowing matured 2D domain knowledge into the 3D segmentation model, which reduces the reliance on laborious 3D annotation work and improves generalization to new categories. However, previous methods use single-view point cloud generation algorithms to bridge the gap between 2D images and 3D point clouds, leaving the incomplete geometry of an object or scene due to occlusions. To address this issue, we propose a novel view synthesis cross-modal few-shot point cloud semantic segmentation network. It introduces the color and depth inpainting to generate multi-view images and masks, which compensate for the absent depth information of generated point clouds. Additionally, we propose a cross-modal embedding network to bridge the domain features between synthesized and original, collected 3D data, and a weighted prototype network is employed to balance the impact of multi-view images and enhance the segmentation performance. Extensive experiments on two benchmarks show the superiority of our method by outperforming the existing cross-modal few-shot 3D segmentation methods.
Crossmodal Few-shot 3D Point Cloud Semantic Segmentation via View Synthesis
Ziyu Zhao Pingping Cai Canyu Zhang Xiaoguang Li Song Wang