Going in the opposite direction with a projection matrix / 3D line from projected 2D point? I have a projection matrix $\textbf{P}$ that projects from 3D coordinates to 2D coordinates. Say I have a 3D point $\textbf{X}(x, y, z)$ that gets projected to the 2D point $\textbf{u}(u, v)$.
Thus, we have that $\textbf{u} = \textbf{PX}$ (in homogenous coordinates).
I can see that it should be possible to use the projection matrix to calculate a 3D line but I'm not sure how to write down the mathematical notation. Any help/pointers would great!


For $\vec x_0 \in \mathbb R^3$ such that $P\vec x_0 = 0$, (i.e. $x_0 \in \ker P$, the kernel or null space of $P$) and any point $\vec x \in \mathbb R^3$ we have that 
$$P(\vec x) = P(\vec x) + 0 = P(\vec x) + P(\vec x_0) = P(\vec x +\vec  x_0).$$
This means that given a projected point $\vec u \in \mathbb R^2$, we can find some $\vec x$ such that $\vec u=P(\vec x)$, but this $\vec x$ is only unique up to elements of the kernel of $P$. If you have $\vec x, \vec y$ such that $P(\vec  x) = P(\vec y)$, then their difference is an element of the kernel of $P$.
This gives us a recipe to calculate the points (this is in fact an affine plane in $\mathbb R^3$) which correspond to a given point $u \in \mathbb R^2$:


*

*Calculate the kernel (null space) of $P$ by solving $P(\vec x_0) = 0$.

*Given $\vec u \in \mathbb R^2$, find any single solution $x$ of $P(\vec x) = \vec u$

*The set you're looking for is $\vec x + \ker P = \{\vec x + \vec x_0:P(\vec x_0)=0\}$. 
In essence, $\ker P$ determines the direction of the line and $x$ determines the intercept.
In general, this doesn't need to be a line, but has the (affine) dimension of the kernel of $P$
