3D rotation orientation My problem involves finding a formula for this:
You're given a line that passes through the origin in 3D space (more specifically, a point on the line). You are then given a point and an angle. The task is to find where the point ends up after a counterclockwise rotation around the line by the angle.
I managed to do this by reformulating the problem as an intersection of three spheres, but I got two results: one clockwise and one counterclockwise.
At this point I have no idea how to figure out which is which, because "counterclockwise" isn't a sufficient definition. You must know which way the line is oriented.
Is there a way to choose from the two destination points such that rotating objects (such as a cube, given its vertices) aren't deformed (because the points rotate in different directions)?

Let's say the line passes through origin $(0,0,0)$ and point $\vec{\ell} =(x_\ell , y_\ell, z_\ell)$, the point to be rotated is $(x, y, z)$.
The line represents our rotation axis. Calculate the unit rotation axis $\hat{a} = ( x_a , y_a , z_a )$:
$$\bbox{ \hat{a} = \frac{\vec{\ell}}{\left\lVert\vec{\ell}\right\rVert} = \left( \frac{x_\ell}{\sqrt{x_\ell^2 + y_\ell^2 + z_\ell^2}} , \frac{y_\ell}{\sqrt{x_\ell^2 + y_\ell^2 + z_\ell^2}} , \frac{z_\ell}{\sqrt{x_\ell^2 + y_\ell^2 + z_\ell^2}} \right ) }$$
To rotate point $\vec{p}$ by angle $\theta$ around unit axis vector $\hat{a}$, we can use Rodrigues' rotation formula:
$$\bbox[#ffffef, 1em]{ \vec{p}^\prime = \vec{p} \cos\theta + (\hat{a} \times \vec{p}) \sin\theta + \hat{a} ( \hat{a} \cdot \vec{p} ) (1 - \cos \theta) }$$
where $\vec{p}^\prime$ is the location of the point after the rotation.
If the $\hat{a}(\hat{a}\cdot\vec{p})(1 - \cos\theta)$ throws you off, note that the first term, $\hat{a}$, is a vector, but both the second and third terms are reals.
In case you have forgotten, for $\vec{a} = ( x_a , y_a , z_a )$ and $\vec{b} = ( x_b , y_b , z_b )$, and $c$ a real,
$$\begin{array}{l|l}
\; & c \vec{a} = ( c x_a , c y_a , c z_a ) \\
\hline
\; & \vec{a} + \vec{b} = ( x_a + x_b , y_a + y_b , z_a + z_b ) \\
\hline
\text{Dot product} & \vec{a} \cdot \vec{b} = x_a x_b + y_a y_b + z_a z_b \\
\hline
\text{Cross product} & \vec{a} \times \vec{b} = ( y_a z_b - z_a y_b , z_a x_b - x_a z_b , x_a y_b - y_a x_b ) \\
\hline
\text{Euclidean length or $\ell^2$-norm} & \left\lVert \vec{a} \right\rVert = \sqrt{ \vec{a} \cdot \vec{a} } = \sqrt{ x_a^2 + y_a^2 + z_a^2 } \\
\end{array}$$
