What is this concept called (differentiating a matrix, NOT talking about Jacobians) In my differential geometry course, we try to prove that the set of orthogonal matrices is a smooth-submanifold. The submersion we use is $f(A) = A^TA - I$, where $I$ is the identity. For $f$ to be a submersion, we need to check that its derivative is surjective. This is the derivative given by my professor:
$$\frac{d}{dX}(A) = \lim_{t\to0}\frac{(A+tX)^T(A+tX) - I}{t}.$$
He said that this is the definition of a matrix derivative, but I don't know any references for this (the Wikipedia page about it would be very helpful, for example).

What you have written is not the derivative $\frac{\partial A}{\partial X}$, which would be zero for a fixed matrix $A$. What you want is $\frac{\partial f}{\partial X} \mid_{A}$, i.e. the derivative of $f$ with respect with $X$, evaluated at $A$. This should be:
\begin{align*}
\frac{\partial f}{\partial X} \mid_{A} &= \lim_{t \to 0} \frac{f(A+tX) - f(A)}{t}\\
& = \lim_{t \to 0} \frac{(A+tX)^{T}(A+tX) - A^{T}A}{t}\\
& = \lim_{t \to 0} \frac{tX^{T}A + tA^{T}X + t^{2}X^{T}X}{t}\\
& = \lim_{t \to 0} X^{T}A + A^{T}X + tX^{T}X\\
& = (X^{T}A + A^{T}X)
\end{align*}
