An Analysis of SVD for Deep Rotation Estimation

Symmetric orthogonalization via SVD, and closely related procedures, are well-known techniques for projecting matrices onto $O(n)$ or $SO(n)$. These tools have long been used for applications in computer vision, for example optimal 3D alignment problems solved by orthogonal Procrustes, rotation averaging, or Essential matrix decomposition. Despite its utility in different settings, SVD orthogonalization as a procedure for producing rotation matrices is typically overlooked in deep learning models, where the preferences tend toward classic representations like unit quaternions, Euler angles, and axis-angle, or more recently-introduced methods. Despite the importance of 3D rotations in computer vision and robotics, a single universally effective representation is still missing. Here, we explore the viability of SVD orthogonalization for 3D rotations in neural networks. We present a theoretical analysis that shows SVD is the natural choice for projecting onto the rotation group. Our extensive quantitative analysis shows simply replacing existing representations with the SVD orthogonalization procedure obtains state of the art performance in many deep learning applications covering both supervised and unsupervised training.


Introduction
There are many ways to represent a 3D rotation matrix. But what is the ideal representation to predict 3D rotations in a deep learning framework? The goal of this paper is to explore this seemingly lowlevel but practically impactful question, as currently the answer appears to be ambiguous.
In this paper we present a systematic study on estimating rotations in neural networks. We identify that the classic technique of SVD orthogonalization, widely used in other contexts but rarely in the estimation of 3D rotations in deep networks, is ideally suited for this task with strong empirical and theoretical support.
A design choice common to all of these models is selecting a representation for 3D rotations. The most frequent choices are classic representations including unit quaternion, Euler angles, and axisangle. Despite being such a well-studied problem, there is no universally effective representation or regression architecture due to performance variations across different applications.
A natural alternative to these classic representations is symmetric orthogonalization, a long-known technique which projects matrices onto the orthogonal group O(3) [23,35]. Simple variations can restrict the projections onto the special orthogonal (rotation) group SO(3) [13,18,44]. This procedure, when executed by Singular Value Decomposition (SVD [9]), has found many applications in computer vision, for example at the core of the Procrustes problem [2,35] for point set alignment, as well as single rotation averaging [11]. A nearly identical procedure is used for factorizing Essential matrices [12].
Despite its adoption in these related contexts, orthogonalization via SVD has not taken hold as a procedure for generating 3D rotations in deep learning: it is rarely used when implementing a model (e.g. overlooked in [21,5,32,26]), nor is it considered a benchmark when evaluating new representations [22,50].
In light of this, this paper explores the viability of SVD orthogonalization for estimating rotations in deep neural networks. Note, we do not claim to be the first to introduce this tool to deep learning, rather our focus is on providing a comprehensive study of the technique specifically for estimating rotations. Our contributions include • A theoretically motivated analysis of rotation estimation via SVD orthogonalization in the context of neural networks, and in comparison to the recently proposed Gram-Schmidt procedure [50]. One main result is that SVD improves over Gram-Schmidt by a factor of two for reconstruction, thus supporting SVD as the preferred orthogonalization procedure.
• An extensive quantitative evaluation of SVD orthogonalization spanning four diverse application environments: point cloud alignment, object pose from images, inverse kinematics, and depth prediction from images, across supervised and unsupervised settings, and benchmarked against classic and recently introduced rotation representations.
Our results show that rotation estimation via SVD orthogonalization achieves state of the art performance in almost all application settings, and is the best performing method among those that can be applied in both supervised and unsupervised settings. This is an important result given the prevalence of deep learning frameworks that utilize rotations, as well as for benchmarking future research into new representations.

Related Work
Optimization on SO (3), and more generally on Riemannian manifolds, is a well-studied problem. Peculiarities arise since SO (3) is not topologically homeomorphic to any subset of 4D Euclidean space, so any parameterization in four or fewer dimensions will be discontinuous (this applies to all classic representations-Euler angles, axis-angle, and unit quaternions). Discontinuities and singularities are a particular nuisance for classic gradient-based optimization on the manifold [37,41].
Early deep learning models treated Euler angle estimation as a classification task [43,38], by discretizing the angles into bins and using softmax to predict the angles. This idea was extended to hybrid approaches that combine classification and regression. In [20], discrete distributions over angles are mapped to continuous angles via expectation and [25] combines classification over quantized rotations with the regression of a continuous offset. In [22] it is shown that typical activations used in classification models (e.g. softmax) lead to more stable training compared to the unconstrained setting of regression. The authors introduce a "spherical exponential" mapping to bridge the gap and improve training stability for regression to n-spheres. All methods that discretize suffer from the problem of increased dimensionality and expressivity. All of the above methods require supervision on the classification objective, which makes them unsuitable for unsupervised settings.
Probabilistic representations have been introduced for modeling orientation with uncertainty [34,8], with the von Mises and Bingham distributions respectively. While these are best suited for multimodal and ambiguous data, such approaches do not reach state of the art in tasks where a single precise rotation must be predicted.
The closest approach to SVD orthogonalization is the recent work of [50] which makes a strong connection between the discontinuities of a representation and their effect in neural networks. In search of continuous representations, they propose the idea of continuous overparameterizations of SO(3), followed by continuous projections onto SO (3). Their 6D representation is mapped onto SO(3) via a partial Gram-Schmidt procedure. This is similar in spirit to SVD orthogonalization which will map a continuous 9D representation onto SO(3) with SVD. We leave the deeper comparison of these two methods to the following sections, where we show that SVD provides a more natural projection onto SO(3) in several respects (Section 3).
SVD derivatives have been presented in [7,30], and there exist multiple works that build neural nets with structured layers depending on components of SVD or Eigendecomposition [16,33,15,3,28]. Close to our setting are [40] which applies the orthogonal Procrustes problem to 3D point set alignment within a neural network, and [17] which proposes singular value clipping to regularize networks' weight matrices.
SVD is amenable for learning via backpropagation. Its derivatives have been presented in [7,30], and there exist multiple works that build neural nets with structured layers depending on SVD or Eigendecomposition [16,33,15,3]. The closest to our setting is [40] which applies the orthogonal Procrustes problem to 3D point set alignment within a neural network, and [17] which proposes singular value clipping to regularize networks' weight matrices. We discuss the stability of SVD orthogonalization in neural networks in the following section.

Analysis
In this section we present our theoretically motivated analysis of SVD orthogonalization for rotation estimation. We start here defining the procedures and introducing well-known results regarding their least-squares optimality before presenting the analysis.
Given a square matrix M with SVD U ΣV T , we consider the orthogonalization and special orthogonalization SVDO is orientation-preserving, while SVDO + maps to SO(n). Orthogonalization of a matrix via SVD is also known as symmetric orthogonalization [23]. It is well known that symmetric orthogonalization is optimal in the least-squares sense [2,13,35]: This property has made symmetric orthogonalizations useful in a variety of applications [44,35,40].
To reiterate, the procedure for deep rotation estimation we will evaluate experimentally in Section 4 is SVDO + (M ) in 3D, which takes a 9-dimensional network output (interpreted as a 3 × 3 matrix M ), and projects it onto SO(3) following Eq. 2.

SVDO(M ) and SVDO + (M ) are maximum likelihood estimates
In this section we show that SVD orthogonalization maximizes the likelihood and minimizes the expected error in the presence of Gaussian noise. Let M = R µ + σN, M ∈ R n×n represent an observation of R µ ∈ SO(n), corrupted by noise N with entries n ij ∼ N (0, 1). Following from the pdf of the matrix normal distribution [10] the likelihood function is The solution is given by SVDO + (M ) (Eq. 3). Similarly, SVDO(M ) will minimize Eq. 4 when R µ ∈ O(n).

Gradients
In this section we analyze the behaviour of the gradients of a network with an SVDO + layer, and show that they are generally well behaved. Specifically, we can consider ∂L ∂M for some a loss function L(M, R) = SVDO + (M ) − R 2 F . We will first analyze ∂L ∂M for SVDO(M ). Letting • denote the Hadamard product, from [16,42] we have T V , we see that X, Y are antisymmetric and X = −Y (this is a result of the loss function depending on U V T ). We can reduce ∂L ∂M = U ZV T where the elements of Z are See Appendix C for the detailed derivation. For SVDO(M ) Eq. 8 tells us ∂L ∂M is undefined whenever two singular values are both zero and large when their sum is very near zero.
For SVDO + (M ), if det(M ) > 0 then the analysis is the same as above. If det(M ) < 0, the extra factor D = diag(1, 1, . . . , −1) effectively changes the smallest singular value s n to −s n . The derivation is otherwise unchanged. In particular the denominator in equation (8) is now s j − s n or s n − s i if either i or j is n. Thus, ∂L ∂M is undefined if the smallest singular value occurs with multiplicity greater than 1. It is large if the two smallest singular values are close to each other, or if they are close to 0.

Error analysis
In this section we approximate the expected error in SVDO(M ) and Gram-Schmidt orthogonalization (denoted as GS(M )) in the presence of Gaussian noise, and observe that the error is twice as large for GS as for SVDO. If M is a matrix with QR decomposition M = QR, define: We consider M = R 0 + σN , a noisy observation of a rotation matrix R 0 ∈ SO(n), where N has i.i.d. Gaussian entries n ij ∼ N (0, 1) and σ is small. Since det(M ) > 0 as σ → 0, the analysis is identical for SVDO + and GS + . The analysis is also independent of R 0 (Appendix A), so for simplicity we set R 0 = I. First we calculate the SVD and QR decompositions of M to first order for N an arbitrary (non-random) matrix.

Proposition 1
The SVD and QR decompositions of M = I + σN are as follows: 1. (SVD) Let N = S + A be the decomposition of N into symmetric and antisymmetric parts. Then, to first order, an SVD of M is given by This corollary is by straightforward calculation since each entry of A and L − L T is Gaussian (see appendix A for the proof). Notably, Gram-Schmidt produces twice the error in expectation (and indeed deviates 1.5 times further from the observation M itself). The same holds for SVDO + and GS + . This difference in performance can be traced to the fact that Gram-Schmidt is essentially "greedy" with respect to the starting matrix, whereas the SVD approach is coordinate-independent.
Although i.i.d. Gaussian noise is not necessarily reflective of a neural network's predictions, it does provide meaningful insight into the relationship between SVDO + and GS + and their relative performance observed in practice. See Appendix A for further remarks.
Proof of Proposition (sketch; see Appendix A for the full proof).
(1) Let M have SVD M = U SV T for some orthogonal matrices U, V and diagonal matrix S ≥ 0.
To first order in σ, we can write (This is using the fact that the antisymmetric matrices give the tangent space to the orthogonal matrices.) The claims come from breaking this equation into symmetric and antisymmetric parts; note if X is (anti-)symmetric and Q is orthogonal, then QXQ T is again (anti-)symmetric. Simplify to get SVDO(M ) = I + σA + O(σ 2 ).
For (2) the proof is similar: we expand the QR decomposition M = QR to first order as Q 0 orthogonal, Q 1 antisymmetric, and R 1 , R 0 upper triangular; examine the upper and lower parts.

Continuity for special orthogonalization
The calculation above shows SVDO(M ) and SVDO + (M ) are continuous and differentiable, at least at M = I. In fact SVDO(M ) is smooth, as is SVDO + except for a discontinuity 1 if (and only if) det(M ) = 0 or det(M ) < 0 and its smallest singular value has multiplicity greater than 1. In fact the optimization problem (3) is degenerate in this case. For example, the 2 × 2 matrix M = diag(1, −1) is equidistant from every rotation matrix; perturbations of M may special-orthogonalize to any R ∈ SO(2). GS + is continuous on a slightly larger domain -det(M ) = 0 -because it makes a uniform choice, negating the n-th column of M if necessary, at the cost of significantly greater error in expectation. This reflects the fact that SVD orthogonalization is coordinate-independent and GS, GS + are not: and similarly for SVDO + . GS and GS + are rotation-equivariant on only one side: is not a function of R 2 and GS(M ); likewise for GS + . See Appendix B for a proof of smoothness and further discussion.

Summary
The results above illustrate a number of desirable properties of SVD orthogonalization. SVDO + is optimal as a projection in the least squares sense as well as in the presence of Gaussian noise (MLE). Viewed through the lens of matrix reconstruction, the approximation errors are half that of the Gram-Schmidt procedure. Finally we show the conditions that lead to large gradient norms (conditions that are rare for small matrices). In the following, we support this theoretical analysis with extensive quantitative evaluations.

Experiments
This section presents the experimental analysis for SVD orthogonalization and numerous baseline methods. To recap, the SVD orthogonalization procedure SVDO + (M ) takes a 9D network output (interpreted as a 3×3 matrix), and projects it onto SO(3) via Eq. 2. The procedure can easily be used in popular deep learning libraries (e.g. PyTorch [31] and TensorFlow [1] both provide differentiable SVD ops), and it does not affect efficiency (3 × 3 SVD adds negligible computational overhead in both forward and backward passes).  For SVD, 6D, 5D, and the classic representations, the loss is (3) this is related to geodesic angle error θ as L(R, R t ) = 2 − 2 cos(θ). All other methods require an additional classification loss. See Appendix D for additional experimental details (architectures, implementations, etc).

3D point cloud alignment
The first experiment is the point cloud alignment benchmark from [50]. Given two shape point clouds the network is asked to predict the 3D rotation that best aligns them. The rotations in the dataset are sampled uniformly from SO(3) (no rotation bias in the data). Table 1 (left) shows geodesic error statistics (mean, median, std) on the test set. We omit the maximum error as it is approx. 180 • for all methods, a reflection of the symmetries in the data rather than limitations of the methods. SVD-Train outperforms all the baselines, and even SVD-Inference performs on par with the best baseline (6D). Interestingly, the hybrid approaches 3D-RCNN and M G underperform the top regression baselines, a point we will return to later. Table 1 (middle) shows the mean errors on the test set as training progresses. The best performing methods at the end of training (SVD variations, and 6D) also show fast convergence. The errors at different percentiles are shown in Table 1 (right).

3D Pose estimation from 2D images
The second experiment follows the benchmark set forth in [22]. Images are rendered from Model-Net10 [45] objects from arbitrary viewpoints. Given a 2D image, the network must predict the object orientation. We used MobileNet [14] to generate image features, followed by fully connected regression layers. Rather than averaging over all 10 ModelNet categories as in [22], we focus on chair and sofa which are the two categories which exhibit the least rotational symmetries in the dataset. Results are shown in Tables 2 and 3. Interestingly, SVD-Inference also performs similarly to SVD-Train on final metrics with faster convergence, indicating short pretraining with SVD-Inference could improve convergence rates.
3D-RCNN and M G are again underperforming the best methods. These hybrid methods have shown state of the art performance on predicting 3D pose from images [20,25], but in those benchmarks the 3D rotations exhibit strong bias (camera viewpoints are not evenly distributed over SO(3)) which is

Pascal3D+
[46] is a standard benchmark for object pose estimation from single images. The dataset is composed of real images covering 12 categories. For training we discard occluded or truncated objects [25,38] and augment with rendered images from [38]. The model architecture is the same as Section 4.2. Table 4 shows results on two categories and the mean over all categories (see Appendix D.7 for results on each of the 12 categories). The individual metrics we report are the median error as well as accuracies at 10 • , 15 • , and 20 • .
The best performing method is clearly S 2 -Reg. As expected, the hybrid method 3D-RCNN performs well on this task, but SVD-Inference and SVD-Train are on par. The SVD variations are also the best performing of the regression methods (those that only train with a rotation loss). Interestingly, SVD-Inference slightly outperforms SVD-Train, which suggests in this scenario where viewpoints have a non-uniform prior, training a network to regress directly to the desired rotation can work well.

Unsupervised rotations
So far we have considered supervised rotation estimation. Given the growing attention to self-or unsupervised 3D learning [48, 26,19,50], it is important to understand how different representations fare without direct rotation supervision. We omit 3D-RCNN, M G , and S 2 -Reg from the experiments below as they require explicit supervision of classification terms, as well as SVD-Inference as it does not produce outputs on SO(3) while training.

Self-supervised 3D point cloud alignment
To begin, we devise a simple variation of the point cloud alignment experiment from Section 4.1. Given two point clouds, the network still predicts the relative rotation. However, now the only loss is L2 on the point cloud registration after applying the predicted rotation. All other experiment details remain the same. From Table 5, SVD-Train performs significantly better than the next closest baseline, 6D.

Inverse kinematics
Our second unsupervised experiment is the human pose inverse kinematics experiment presented in [50]. A network is given 3D joint positions and is asked to predict the rotations from a canonical "T-pose" to the input pose. Predicted rotations are transformed back to joint positions via forward kinematics, and the training loss is on the reconstructed joint positions. We use the code provided by [50]. Table 6 presents the results. SVD-Train shows the best overall performance while 6D is closer than in other experiments.

Unsupervised depth estimation
The final experiment considers self-supervised learning of depth and ego-motion from videos [48]. Given a target image and source images, the model predicts a depth map for the target and camera poses from the target to sources. Source images are warped to the target view using predicted poses, and reconstruction loss on the warped image supervises training. In [48] the rotational component is parameterized by Euler angles. Following [48], we report the single-view depth estimation results  Table 7). The error metrics are in meters while accuracy metrics are percentages up to a distance threshold in meters (see [4] for a description).
Observe that the difference between the best and second best method in each metric is small. This is not surprising since the camera pose is a small (albeit important) part of a complex deep architecture. Nonetheless, SVD-Train performs best for 4 out of the 7 metrics, and second best in another two. For driving data the motion is likely to be mostly planar for which axis-angle is well suited. Finally, it is worth noting that carefully selecting the rotation representation is important even in more complex models -the default selection of Euler angles in [48] is outperformed in every metric.

Conclusion
The results of the previous sections are broad and conclusive: a continuous 9D unconstrained representation followed by an SVD projection onto SO (3) is consistently an effective, and often the state-of-the-art, representation for 3D rotations in neural networks. It is usable in a variety of application settings including without supervision. The strong empirical evidence is supported by a theoretical analysis that supports SVD as the preferred projection onto SO(3).
[  Note that since SVDO(RM ) = R · SVDO(M ) and GS(RM ) = R · GS(M ) for any orthogonal matrix R, and likewise for SVDO + , GS + if R is special orthogonal. Therefore the error analyses are the same for all matrices M : since orthogonal matrices preserve Frobenius norm and R −1 N has the same distribution as N since N was assumed isotropic.
with U 0 , V 0 orthogonal, U 1 , V 1 antisymmetric and Σ 0 , Σ 1 ≥ 0 diagonal. This is using the fact that the antisymmetric matrices give the tangent space to the orthogonal matrices. Similarly, the tangent space to the diagonal matrices is given again by the diagonal matrices. This gives an overall expression for M as Setting σ = 0 we see I = U 0 Σ 0 V T 0 , which implies Σ 0 = I and U 0 = V 0 . Next, collecting the first-order σ terms gives If a matrix X is (anti-)symmetric and Q is orthogonal, then QXQ T is again (anti-)symmetric. So, the symmetric and antisymmetric parts of the equation are Note that the first equation is an SVD of the symmetric part of N , while the second equation shows that U 1 and V 1 satisfy U 1 + V T 1 = U T 0 AU 0 . Finally, dropping the Σ 0 + σΣ 1 factor from Eq. (19) and expanding out shows that SVDO(I + σN ) = I + σA + O(σ 2 ).
(2) Let M = QR, where Q is orthogonal and R is upper-triangular with positive diagonal. As above, by expanding to first order in σ we have with Q 0 orthogonal, Q 1 antisymmetric, and R 1 , R 0 upper triangular. Setting σ = 0, we see I = Q 0 R 0 and so Q 0 = R 0 = I. For the σ terms, we split N into its upper, lower and diagonal parts to get which by comparing parts gives Q 1 = L−L T and R 1 = U +D+L T . Then GS(M ) = I +σ(L−L T ) by simple algebra.
We now prove Corollary 1. N (0, 1), then with error of order O(σ 3 ), Proof. Simplifying the error expressions using the first-order calculations in the Proposition gives with notation for S, A, U, D, L as in the proposition. Thus each expression is σ 2 times the Frobenius norm of the corresponding matrix. Each entry of A, L−L T , S and U +D+L T is a linear combination of the entries of N , hence is Gaussian since N has i.i.d. Gaussian entries n ij ∼ N (0, 1). The expectations are the sums of the entrywise expectations of these matrices. For example, A = 1 2 (N − N T ) has six nonzero entries of the form 1 2 (n ij − n ji ), each having variance 1 2 , so E[ A 2 F ] = 3. For L − L T , the above diagonal entries are −n ji and the below-diagonal entries are n ij , and the diagonal is 0, so the total variance is 6. The other two calculations are similar (the entries do not all have the same variances).
Remark. The tangent space to the identity matrix along the orthogonal matrices is the space of antisymmetric matrices. Both of the calculations above can be thought of as giving orthogonal approximations of the form where A ′ is a choice of antisymmetric matrix that depends on the approximation method. The fact that SVDO(M ) produces the approximation A ′ = A = 1 2 (N − N T ) means it corresponds to the natural projection of N onto the orthogonal tangent space. By contrast, GS(M ) produces A ′ = L − L T , essentially a "greedy" choice with respect to the starting matrix (minimizing the change to the leftmost columns). For certain matrices GS can have smaller error: for example if N happens to be upper-triangular, GS(M ) = I and the error is zero. For isotropic noise, however, the SVD approximation is the most efficient in expectation.

A.1 Accuracy of error estimates as σ increases
From Corollary 1 (Sec 3.3) we see special-orthogonalization with Gram-Schmidt (GS + ) produces twice the error in expectation as SVD (SVDO + ) for SO(3) reconstruction when inputs are perturbed by Gaussian noise. We compare these derived errors with numerical simulations. See Figure 1. We calculate the derivative with respect to Q ∈ O(n): for A an antisymmetric matrix, If this vanishes for every A, then M T Q is symmetric, that is, (M, Q) is a root of the function g(M, Q) = M T Q − Q T M . Let M 0 be a fixed matrix. As discussed above, the optimal solution to this equation is given by an SVD, To show that Q is a smooth function of M , it suffices by the Implicit Function Theorem to show that the Jacobian matrix ∂g ∂Q is full-rank at (M 0 , Q 0 ). To see this, we differentiate it again: where A is antisymmetric. Some algebra shows that this is, equivalently, To see that this is an invertible transformation of A, note that conjugating by V 0 is invertible since V 0 is orthogonal. So it is equivalent to show that the function is invertible. This function just rescales the entry a ij to (s i + s j )a ij . Since the singular values are positive this is invertible as desired.

Proposition 3 The special symmetric orthogonalization is a smooth function of M if either of the following is true:
• det(M ) > 0, • det(M ) < 0 and the smallest singular value of M has multiplicity one.
Proof sketch. The analysis is identical to the main proof, except that if det(M ) < 0, S 0 is effectively altered so that the last entry is changed from s n to −s n . Thus the function A → S 0 A + AS 0 now sends the entry a ij to (±s i ± s j )a ij , with negative signs at i = n and/or j = n. If s n occurred with multiplicity one, the result is still invertible since s i − s n = 0 for i = n and for i = j = n the coefficient is −2s n . Otherwise, however, s n−1 = s n and the operation sends a n−1,n to (s n−1 − s n )a n−1,n = 0; likewise for a n,n−1 . In this case there are many optimal special orthogonalizations of M 0 , and the operation is not even continuous in a neighborhood of M 0 .

C Gradients
Here we provide the detailed derivation sketched out in Sec 3.2. We will first analyze ∂L ∂M for SVDO(M ). With • denoting the Hadamard product, from [16,42] we have Inspecting the individual elements of (F • X)Σ and Σ(F • X)) we have For SVDO(M ) Eq. 40 tells us ∂L ∂M is undefined whenever two singular values are both zero and large when their sum is very near zero.
For SVDO + (M ), if det(M ) > 0 then the analysis is the same as above. If det(M ) < 0, the extra factor D = diag(1, 1, . . . , −1) effectively changes the smallest singular value s n to −s n . The derivation is otherwise unchanged. In particular the denominator in equation (40) is now s j − s n or s n − s i if either i or j is n.

C.1 Gradients observed during training
In Figure 2 (left) we see the gradient norms observed while training for point cloud alignment (Section 4.1). SVD-Train has the same profile as for 6D (GS + ). SVD-Train converges quickly (relative to all other methods) in all of our experiments, indicating no instabilities due to large gradients.
On the right of Figure 2 we profile the gradients for the scenario where we begin training with the SVD-Inference loss and switch to SVD-Train after 100K steps (after roughly 4% of training iterations). SVD-Inf trains the network to produce outputs that are close to SO(3), which eliminates some conditions of instability in Eq. 40. This is confirmed by seeing much smaller gradient norms after switching to SVD-Train at 100K steps. Note, this approach was never used (or needed) in our experiments.

D.1 Additional baseline implementation details
• 6D is the partial Gram-Schmidt method which computes GS + (M ). Our implementation follows exactly the code provided by [50]. • 5D maps outputs in R 5 onto SO(3) as described in [50].
• Spherical Regression (S 2 -Reg) regresses to n-spheres. Following [22], we use regression to S 1 for Pascal3D+ and S 3 regression (quaternions) for their ModelNet experiments (section 4.2). The method combines abs. value regression with sign classification. Our implementation of the final regression layers follows the provided code. We select the hyperparameter that balances the classification and regression losses by a simple line search in the neighborhood of the default provided in [22].  . We found this version of Geodesic-Bin-and-Delta outperformed the One-delta-network-per-pose-bin variation in these experiments. For Pascal3D+ we follow the reference and use K-means with K = 200. This method also requires both classification and regression losses and we were unable to train successfully in the unsupervised setting. • Quaternion, Euler angles, and axis-angle are the classic parameterizations. In each case they are converted to matrix form before the loss is applied. In our experiments we did not filter any outputs from the network representing angles (e.g. clipping values or applying activations such as tanh). We found this gave the best results overall.

D.2 Learning rate decay
An observation from the point cloud registration results is that the curves for mean test errors as training progresses do not decay smoothly as one might expect for any method (Table 1, middle, in the main paper). This is in part due to the training code from [50] does not utilize a learning rate decay for this experiment. It is reasonable to observe the variance in evaluation when a decay is introduced as would be common in practice. Table 8 (left) shows the curves when the learning rate is exponentially decayed (decay rate of 0.95, decay steps of 35K). The evaluation over time is smoother but the results are consistent with those presented in the main paper.

D.3 Geodesic loss
In [50] it was shown that geodesic loss for training did not alter the results much, and we have the same observation. The model architecture follows exactly the architecture described in [50]. Point clouds are embedded with simplified PointNet (4-layer MLP) ending with a global max-pooling. Three dense layers  The regression layers are the same as above. The image embeddings are generated with Mo-bileNet [14]. We found no difference in performance between MobileNet and VGG16 [36] (not surprising given the comparison in [14]).

D.5 Inverse kinematics (Sec. 4.4.2) and unsupervised depth estimation (Sec. 4.4.3)
For these experiments the training and evaluation code is provided by [50] and [48], respectively. We simply change the rotation representation layer.

D.6 Remark on timings
We noticed no measurable difference in training time with SVDO + as expected, since the forward and backward pass for small, e.g. 3 × 3, SVD is efficient.

D.7 Pascal3D+ full results.
Here in Table 9 we show the results for all 12 categories in the Pascal3D+.