The Adam optimizer [1]} is utilized with weight decay 1e-4 and the initial learning rate is set as 5e-5. The model is trained for 50 epochs by default. We set the point cloud sub-model size \(N_{s}\)  as 2048 and set \(N_{\delta } = N/N_{s} + 1\) . The projected images with the resolution of 1920\(\times \) 1080\(\times \) 3 are randomly cropped into image patches at the resolution of 224\(\times \) 224\(\times \) 3 as the inputs. During the experiment, 6 sub-models and 4 projected images are selected at random for each point cloud. The PointNet++ [2]} is utilized as the point cloud encoder and the hierarchical ResNet50 [3]} is used as the image encoder, where the ResNet50 is initialized with the pre-trained model on the ImageNet database [4]}. The multi-head attention module employs 8 heads and the feed-forward dimension is set as 2048. The weights \(\lambda _{1}\)  and \(\lambda _{2}\)  for \(L_{MSE}\)  and \( L_{r a n k}\)  are both set as 1.
