Q:

How to quantify the distance between matrices with an irrelevant rotation factor?

Suppose you have two invertible matrices $A$, $B$ in $\mathbb{R}^{n\times n}$, that is, $A,B\in GL(n)$.  You want to define a distance between them that ignores arbitrary rotational factors, so basically if $M\in O(n)$ is an orthogonal matrix and $A = MB$, $||A-B|| = 0$.  One way to do this would be to take the geodesic length between $A, B \in GL(n)/O(n)$, the quotient group of invertible matrices over orthogonal matrices.  How would you compute this geodesic distance?

A:

Generally, the distance given by minimizing length in a Riemannian metric does not lend itself to actual computations. It's more of a thing to prove theorems about. But this case is somewhat manageable. 
If you ignore rotation only on the left ($A\sim UA$ for $U\in O(n)$), then the appropriate tool is the polar decomposition $A=UP$ where $P=\sqrt{A^TA}$ is a positive definite matrix. By definition, the matrix $P$ eliminates any orthogonal factors attached to $A$ on the left. It remains to define a notion of distance between positive definite matrices $P_1$ and $P_2$. You could measure the additive difference $\|P_1-P_2\|$, or multiplicative $\log \max(\|P_1^{-1}P_2 , P_1P_2^{-1}\|)$; in both cases $\|\cdot \|$ can be any matrix norm (for the second case, it should be normalized so that the identity matrix has norm $1$). The choice depends on what you want to do with this distance.  
The above distance functions are metrics, but they most likely do not come from a Riemannian metric. If you really want a distance that is based on a Riemannian metric, see this answer by Robert Bryant. It is stated for  $\mathrm{SL}$, but since $\mathrm{GL}_+$ is just the product $\mathrm{SL}\times \mathbb R_+$,  it might works the same:
$\det(tP_1-P_2)=\det P_1\prod_{j=1}^n (t-\lambda_j)$, and then $d(P_1,P_2)=\sqrt{\sum ( \log \lambda_j)^2}$. 

Alternatively,  if you ignore rotation both on left and right ($A\sim UA\sim AU$ for $U\in O(n)$), then the appropriate tool is the singular value decomposition $A=U\Sigma V^*$ where $\Sigma$ is obtained by diagonalizing $ \sqrt{A^TA}$. In other
words, the diagonal of $\Sigma $ consists of the singular values $\sigma_i$ of $A$, the square roots of the eigenvalues of $A^TA$. 
The group of diagonal matrices with positive diagonal entries is just a power  of $\mathbb R_+$, and the natural metric on it is 
$\sqrt{\sum (\log (\sigma_i/\sigma_i'))^2}$. 

