Q:

Find if a point lies within the angle formed by three other points

I have three points (A, B, C) in a 3d space that define an angle like this:

I need a fast way to know if another point D (that I already know is coplanar with the plane ABC) lies within the angle formed by <ABC.

A:

You can express vector AD as a linear combination of AB and AC.  If the coefficients of the linear combination are both positive, then D is inside the angle.
So basically, you are solving the system of equations (3 equations, 2 unknowns):
$
\left[\begin{array}{rr|r}
b_x-a_x & c_x-a_x& d_x-a_x \\
b_y-a_y & c_y-a_y& d_y-a_y \\
b_z-a_z & c_z-a_z& d_z-a_z \\
\end{array}\right]
$
which represent the following equations:
$
(b_x-a_x) \times I + (c_x-a_x) \times J = d_x-a_x \\
(b_y-a_y) \times I + (c_y-a_y) \times J = d_y-a_y \\
(b_z-a_z) \times I + (c_z-a_z) \times J = d_z-a_z \\
$
and if your solution vector has positive components ($I>0, J>0$), then D is inside that angle.  If there is no solution, then D is not in the ABC plane.

