OMAD: Object Model with Articulated Deformations for Pose Estimation and Retrieval

Articulated objects are pervasive in daily life. However, due to the intrinsic high-DoF structure, the joint states of the articulated objects are hard to be estimated. To model articulated objects, two kinds of shape deformations namely the geometric and the pose deformation should be considered. In this work, we present a novel category-specific parametric representation called Object Model with Articulated Deformations (OMAD) to explicitly model the articulated objects. In OMAD, a category is associated with a linear shape function with shared shape basis and a non-linear joint function. Both functions can be learned from a large-scale object model dataset and fixed as category-specific priors. Then we propose an OMADNet to predict the shape parameters and the joint states from an object's single observation. With the full representation of the object shape and joint states, we can address several tasks including category-level object pose estimation and the articulated object retrieval. To evaluate these tasks, we create a synthetic dataset based on PartNet-Mobility. Extensive experiments show that our simple OMADNet can serve as a strong baseline for both tasks.

: Our OMAD representation for modeling articulated objects. In OMAD, we use shape parameters β β β to capture shape geometric deformations and use joint states Θ Θ Θ to capture pose deformations. We use to represent the variables in canonical space (e.g.P P P ), which distinguishes from variables in camera space (e.g. P P P).
propose an explicit parametric representation to depict shape and pose of articulated objects in the same category. There are two sources of articulated deformations, namely shape geometric deformation on parts and pose deformations on joints. To handle both deformations, we propose a novel Object Model with Articulated Deformations (OMAD) (Fig. 1). To describe the shape geometric deformations, given an articulated object, we consider its rest structure (i.e. shape geometry, joint parameters) in the axis-aligned, zero-centered canonical space. Then for its shape, which is represented by ordered sparse keypoints, can be derived from the proposed shape function P P P = S(β β β ; B B B), where β β β is a pose-invariant shape parameters in canonical space, and B B B is the category-specific linear shape basis. For the joint parameters Φ Φ Φ , we devise a non-linear joint function Φ Φ Φ = J(β β β ; Γ Γ Γ) with parameters Γ Γ Γ. The parameters {B B B, Γ Γ Γ} of shape function and joint function can be regarded as categoryspecific priors, and can be learned from existing object dataset like PartNet-Mobility [28] with proposed OMAD-PriorNet. In this way, the shape geometric deformation can be fully described by (P P P , Φ Φ Φ ). On the other hand, the pose deformation on joints can simply defined by its joint state Θ Θ Θ in camera space.
To combine both deformations, we devise a deformation function W (P P P , Φ Φ Φ , Θ Θ Θ). It can transform the rest structure in canonical space to camera space. In this sense, we can depict arbitrary articulated object of a known category simply with β β β and Θ Θ Θ. To predict β β β and Θ Θ Θ, we propose a novel OMADNet, which is mainly constituted by a neural prediction stage for coarse estimation of β β β and Θ Θ Θ and an optimization stage to refine the prediction results. To show the applicability of the proposed OMADNet, we apply it into two different tasks, namely articulated object pose estimation and articulated object retrieval. The predicted shape parameters β β β can be used for pose-oblivious object retrieval [8]. Besides, the predicted joint state Θ Θ Θ and joint parameters Φ Φ Φ can directly address the object pose estimation task. To evaluate our method, we select articulated object models of 5 categories from PartNet-Mobility [28] to create a dataset named ArtImage. For the articulated object pose estimation task, we achieve better results on joint state error and joint distance error compared to existing method. For the object retrieval task, we achieve 82.3 average mAP on 5 categories.
Our contribution can be summarized as follows: • A novel category-specific articulated modeling approach named OMAD, which ex-plicitly models both shape deformation and pose deformation at category-level.
• With the proposed OMAD, we provide a simple OMADNet to estimate the shape parameters and joint states required to describe the articulated object from its single observation. The output of OMADNet can be used to address the category-level object pose estimation task and the articulated object retrieval task.

Related Work
Articulated Object Shape Signature and Retrieval. The shape parameters in our proposed OMAD which capture the geometric features of articulated objects and summarize them into a vector are mostly like the shape signature adopted in [2,8,12,13,18,19]. However, these shape signatures cannot be directly applied to our case for at least two reasons: (1) Their methods are largely relied on the mesh representation, such as spectral-based [12], graphbased [2], geodesic-based [8], SDF-based [18], while our shape is in point cloud format. (2) None of the previous methods are interested in the joint parameters of the articulated objects, while our representation takes both joint states and joint parameters into account. For articulated object retrieval task, the standard benchmarks are the the McGill shape benchmark (MSB) [22] and ISDB dataset [8]. However, the object models for a certain category are very few for both datasets, which cannot support the training of our proposed OMAD-PriorNet and OMADNet. Low-rank Shape Prior. Our proposed OMAD adopts a low-rank shape prior based approach to model the sparse shape. Such choice has been proved effective in many tasks, such as instance segmentation [23], NRSfM [3,4,14,25], object structure learning [7], human body modeling [17,21]. To the best of our knowledge, we are the first to apply it to the generic articulated object modeling. It is worth noting that our method differs from SMPL [17] and its variants [21] from at least two aspects: (1) SMPL [17] relies on handcrafted mesh templates with fixed topology and vertex number, while our OMAD only requires flexible point cloud input.
(2) SMPL [17] performs PCA on standard meshes to obtain shape basis, while our proposed OMAD-PriorNet can learn shape basis in an unsupervised manner directly from raw point cloud.
Category-level Articulated Object Pose Estimation. Cateogry-level object pose estimation task aims at predicting poses of unseen objects in the same category, which is firstly introduced by NOCS [27]. A-NCSH [15] extends this task to articulated objects. Later, several works on this task are conducted on videos [11,16]. Our proposed OMADNet can address the category-level articulated object pose estimation task given a single depth image as input. Overall, our work is closely related to [15] with significant different on methodology. Firstly, [15] takes a part-centered perspective, the joint states are estimated from the predicted part segmentations and poses. Our method takes a joint-centered perspective, the joint states are directly predicted by OMADNet. Secondly, [15] uses dense NOCS coordinates to represent articulated objects, while our method use ordered keypoints to obtain sparse correspondence, which largely reduces optimization complexity. Thirdly, Li et al. [15] models the shape deformation implicitly with segmentation and NOCS coordinates, which cannot explicitly reflects the complete shape deformation by parameterization as our proposed OMAD representation. Thus it cannot support other important tasks like object retrieval.

Object Model with Articulated Deformations, OMAD
In this section, we will give a definition about the articulated objects which the OMAD can support (Sec. 3.1). As the representation of pose deformation on the joints are self-evident, we will mainly discuss how to model shape geometric deformations in canonical space(Sec. 3.2) with shape function (Sec. 3.2.1) and joint function (Sec. 3.2.2). A general articulated object in camera space is formed by transforming the keypoints and joint parameters in canonical space via a deformation function (Sec. 3.3).

Object Articulation
We take a joint-centered perspective to study articulated objects. Kinematic tree is one kind of abstraction of articulated objects (See Fig. 2). We denote the joint that connects one node and its parent node in the kinematic tree as the reference joint of this node. For example, if joint k connects node a and its parent node b, then joint k is the reference joint of node a.
Here we consider three types of joints: free joint, revolute joint and prismatic joint (See Fig. 2). (1) Free joint is the joint that connects the world node with the base part (root node) in the kinematic tree. The joint state of free joint can be defined as θ θ θ ( f ree) = (R ( f ) , t ( f ) ), which can be seen as the 6D pose of base part in the world coordinate space. (2) For revolute joint, joint parameters can be defined as φ φ φ (r) = (u u u (r) , q q q (r) ), where u u u (r) represents the direction of joint axis and q q q (r) represents the pivot point of joint axis. The joint state is denoted by θ θ θ (r) , which is the relative rotation angle compared to a pre-defined rest state in canonical space. (3) For prismatic joint, joint parameters can be defined as φ φ φ p , where u u u (p) represents the direction of joint axis. The joint state is denoted by θ θ θ (p) , which is the relative distance along the joint axis compared to a pre-defined rest state in canonical space.
With the introduction of reference joint and free joint, the part number and the joint number can be equal. Given an articulated object with K joints, the overall joint state is denoted as Θ Θ Θ = {θ θ θ 1 , . . . , θ θ θ K }.

Geometric Deformation in Canonical Space
As pose deformation can be independently considered by Θ Θ Θ, we model the shape geometric deformation in the canonical space (See Fig. 3). Objects of the same category in the canonical space have the following properties: (1) The directions in the joint parameters are aligned. (2) The whole object is zero-centered. (3). The joint states are the same and predefined. To note, our canonical space does not normalize the object scale, because object scale is a crucial part of the shape geometry and can be explicitly modeled by the OMAD representation. We use to represent the variables in canonical space (e.g.P P P ), which distinguishes from variables in camera space (e.g. P P P). As the articulation is associated with shape and joint, thus we model the geometric deformation with a shape function and joint function.

Shape Function
As detailed geometric deformation is hard to be modeled, we adopt the sparse shape representation, which seeks sparse keypoint correspondences among objects from the same category. Inspired by [7], we predict the sparse shape with the low-rank shape prior. Given a complete object point cloud in canonical space, the sparse shape P P P with M keypoints can be generated by shape parameters β β β using shape function S(·): be the matrix of all such shape basis, which will be shared in the same category.

Joint Function
Given the rest-state object in canonical space, we can define joint parameters φ φ φ = (u u u , q q q ), u u u ∈ R 3 , q q q ∈ R 3 for each joint. As the corresponding joint direction u u u is aligned in the canonical space, and the pivot point q q q for each joint are strongly correlated to the shape geometry, we propose to predict joint parameters in canonical space from shape parameters β β β . All the joint parameters Φ Φ Φ = {φ φ φ 1 , . . . , φ φ φ K } of the K joints can be generated by joint function J(·): where Γ Γ Γ is the parameters of the joint function J(·). Since the relationship between shape parameters and joint parameters in canonical space are highly non-linear, we design a twolayer MLP with ReLU activation as J(·).

Deformation function
To combine both shape geometric deformation and pose deformation, we introduce a deformation function P P P = W (P P P , Φ Φ Φ , Θ Θ Θ), which can transform the sparse shape P P P in zero-centered canonical space with joint parameters Φ Φ Φ according to joint states Θ Θ Θ to the observed sparse shape in the camera space P P P. Each keypoint p p p i in P P P that belongs to part k is transformed into p p p i in P P P according to Eq. 3 and Eq. 4:

Learning the OMAD Prior
The parameters of shape function S(β β β ; B B B) and joint function J(β β β ; Γ Γ Γ) are category-specific, and we call these parameters {B B B, Γ Γ Γ} as OMAD prior. We design a network called OMAD-PriorNet to learn OMAD prior for each cateogry in an unsupervised manner. The parameters {B B B, Γ Γ Γ} of shape function S(β β β ; B B B) and joint function J(β β β ; Γ) will be treated as shared network parameters of OMAD-PriorNet for each category, and they will be updated during training process. After the training, these parameters will be fixed as category-specific OMAD prior, then we will predict shape parameters β β β and keypoints P P P for each instance in the training set with the pre-trained OMAD-PriorNet which will be used as ground truth in the training of OMADNet described in Sec. 4.1. Please refer to the supplementary materials for all the loss functions and network structure of OMAD-PriorNet.
Learning of Shape Function. To learn the shape basis B B B for shape function S(·), we adopt a similar design in [7]. We make some modifications on the loss functions to adapt to the articulated objects. The tweaks on loss functions are listed in the following: (1) We add an additional L 2 regularization loss on β β β which can improve the generalization ability. (2) The original chamfer loss, coverage loss and surface loss is changed to part-level. (3) We add a separation loss to ensure the keypoints not collapsed into a cluster.
Learning of Joint Function. The parameters Γ Γ Γ of joint function J(·) is optimized during training according to the proposed joint loss defined in Eq. 5.
where L µ µ µ is a cosine-similarity loss that supervises joint direction and L q q q is the L 2 loss between the predicted joint pivot point and the ground-truth joint pivot point.

OMADNet
With the OMAD representation and the pre-learned OMAD priors, given a point cloud X X X = x x x i ∈ R 3 | i = 1, . . . , N reprojected from a depth image, we propose a OMADNet to predict the shape parameter β β β and joint states Θ Θ Θ. The overall pipeline is shown in Fig. 4.

Coarse Prediction of Shape Parameters and Joint States
Feature Extractor. We adopt PointNet++ [20] to extract d-dimensional dense point-wise feature F dense ∈ R N×d from X X X. Then we can generate global feature F global ∈ R d using a max pooling layer with the extracted dense feature. The Shape Branch. The shape branch learns to predict pose-invariant shape parameters β β β in canonical space. It uses a two layer MLP with global feature F global as input to regress a vector as shape parameters β β β . We supervise β β β by Eq. 6: where β β β * is obtained by the pre-trained OMAD-PriorNet described in Sec. 3.4, and L joint is defined in Eq. 5. L β β β indirectly supervises β β β by generating keypoints P P P = S(β β β ; B B B) and joint parameters Φ Φ Φ = J(β β β ; Γ Γ Γ) in canonical space.The parameters {B B B, Γ Γ Γ} are obtained from pre-trained OMAD-PriorNet and are fixed during training. The Joint State Branch. This branch predicts joint states using a two layer MLP with global feature F global as input to regress joint states Θ Θ Θ. As there are three joint types in consideration, we treat them separately. For the free joint, θ θ θ base = {R R R base ,t t t base }, we directly supervise R R R base , which is represented by a quaternion vector, via a cosine-similarity loss L base(R) while the loss L base(t) for t t t base is a Root Mean Square (RMS) loss. For the 1D revolute (L r ) or prismatic (L p ) joint, we use L 2 loss on θ θ θ (r) or θ θ θ (p) . The overall loss for joint state is: where 1 (·) j indicates whether it is the joint type for the joint j. The Keypoint Branch. The keypoint branch will directly predict ordered keypoint targetŝ P P P ∈ R M×3 in camera space using dense feature F dense . These keypoints are for the optimization process described later in Sec. 4.2. Similar to PVN3D [10], we predictP P P with an offset-voting mechanism, that is every point will regress all the M keypoint offsets with respect to its own location. These offsets are denoted by O ∈ R N×M . However, it is hard to directly regress keypoint offsets in camera space for articulated objects due to large joint deformations and self-occlusion. So we additionally predict an attention map S of input point cloud X X X to focus areas around keypoints. Specifically, the j-th keypointp p p j ∈P P P is generated by Eq. 8:p where s i, j ∈ [0, 1] is the attention score of i-th point for j-th keypoint, o o o i,k ∈ O is the 3D offset of j-th keypoint repective to the i-th point, x x x i ∈ X X X is the 3D coordinates of i-th input point. The prediction ofP P P is supervised by: where Θ Θ Θ * is the ground truth of the joint state, the rest of deformation function W (·) is obtained from pre-trained OMAD-PriorNet in Sec. 3.4.

Optimization-based Estimator
Since the regression of neural network is known to be noisy, empirically the estimated highlevel properties like shape parameters β β β and joint states Θ Θ Θ are less accurate than the predicted ordered keypoint targetsP P P in camera space. Thus, to push the limits of the prediction results in inference, we propose to refine β β β and Θ Θ Θ with an energy function E defined in Eq. 10: where P P P is the camera-space keypoints generated by deformation function W (·). The energy function E is differentiable, so we can try to find optimal shape parameters β β β and joint states Θ Θ Θ to fit the predicted keypoint targetsP P P during the optimization process. The initial values of β β β and Θ Θ Θ are obtained from OMADNet prediction, which vastly reduces the optimization time. Please refer to the supplementary files for further details.

Dataset
To the best of our knowledge, there is no public image dataset for articulated object tasks in category level. So we create a synthetic dataset named ArtImage to evaluate OMADNet. We select 5 categories of object models from PartNet-Mobility dataset [28] and render depth images under random joint states, object positions and viewpoints for each category with Unity [1]. We will make our dataset public in the future. Please refer to the supplementary materials for further details of ArtImage.

Experiments on Category-level Pose Estimation
Metrics. We use the following metrics to evaluate the performance of our algorithm on category-level pose estimation. (1) joint state. All the joint states error are evaluated in camera space, including mean angle error for revolute joints (max error 90 • for missing ones) and mean distance error for prismatic joints (max error 1 for missing ones). (2) joint parameters. All the joint parameters are evaluated in camera space. we use mean angle error to evaluate the orientation of joint axis, and use mean distance error to evaluate the distance between two joint axes.
Main Results. Table 1 shows the performance of different methods on category-level pose estimation. We can see that OMAD(refine) surpasses OMAD(initial) by a large margin, which proves the effectiveness of our optimization-based estimator. Compared to state-ofthe-art method A-NCSH [15], our OMAD(refine) achieves better results on joint state error and joint distance error on most categories. It is worth noting that the prediction of A-NCSH [15] heavily relies on part segmentation, which could result in missing parts or joints when segmentation fails. While our method could utilize kinematic prior information to Table 1: Performance comparison on category-level pose estimation task on ArtImage. OMAD(initial) represents the initial joint state and joint parameters predicted by OMAD-Net, and OMAD(refine) represents the refined results with our optimizer. prevent missing joints. Besides, our method is much faster than A-NCSH because we adopt sparse keypoints for optimization. We show some qualitative results of our OMAD(refine) in Fig. 5. We can see that the predicted keypoints after optimization can achieve semantic consistency in category, and maintain object structure under heavy self-occlusion.
Ablation Study. Table 2 shows some ablation experiment results. We use optimizationbased estimator as default setting in all the ablation experiments. We can see from Table 2 that the use of attention score vastly boosts the performance on all metrics when predicting keypoint targets. We also analyze the upper bound of our method in Table 2. The overall performance improves a lot when we replace keypoint targetsP P P predicted by keypoint branch with ground-truth keypoints during optimization process. This indicates that our OMAD representation has great potential and the bottleneck of our method is the quality of predicted keypoint locations in camera space. Please see supplementary files for ablation study on keypoint number, shape parameters β β β and experiment results on real images.

Experiments on Articulated Object Retrieval
Metrics. The pose-invariant shape parameters predicted by OMADNet can be used as shape signatures for articulated object retrieval task. We use the Euclidean distance of shape parameters to estimate the similarity between object instances. We randomly choose 100 depth images as test query set for each category. We report the mAP (mean Average Precision) of top-10 instances on five categories in the test dataset.  Results. To the best of our knowledge, there is no available retrieval method specially designed for our setting. Inspired from [5], we design a simple but effective Siamese Network as baseline which uses PointNet++ [20] backbone and contrastive loss [9] for training. Please see supplementary files for further details of the Siamese Network. Table 3 shows some quantitative results. Although our method are not specially designed for retrieval task, we still achieve better performance compared to the baseline on most categories. We find that the Siamese Network can not distinguish some similar instances in drawer category, which vastly harms the performance, while our method can distinguish them very well. Fig. 6 shows t-SNE [26] visualization of shape parameters predicted by OMADNet. We can see that the predicted shape parameters can distinguish different instances well and maintain pose-invariant.

Conclusion
In this work, we propose a novel category-level representation, OMAD, for articulated object by modeling the two kinds of deformations namely the shape geometric deformation and pose deformation. The OMAD can describe articulated objects with shape parameters and joint states. Then for a naturally observed object, we propose to predict such OMAD information via OMADNet. The predicted OMAD information can be used for multiple tasks, such as category-level object pose estimation and articulated object retrieval.