differentiation of a matrix function In statistics, the residual sum of squares is given by the formula 
$$ \operatorname{RSS}(\beta) = (\mathbf{y} - \mathbf{X}\beta)^T(\mathbf{y} - \mathbf{X}\beta)$$
I know differentiation of scalar functions, but how to I perform derivatives on this wrt $\beta$? By the way, I am trying to take the minimum of RSS wrt to $\beta$, so I am setting the derivative equal to 0. 

I know somehow product rule has to hold. So here I have the first step 
$$-\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta) + (\mathbf{y}-\mathbf{X}\beta)^T(-\mathbf{X})= 0$$

I wouldn't do this by differentiating.  Let $P$ be the matrix that projects orthogonally onto the column space of the design matrix $X$.  Then $y-Py$ is orthogonal to $P\mathbf y - X\beta$, since $P\mathbf y$ and $X\beta$ are both in the column space of $X$.  Then we have
\begin{align}
\operatorname{RSS}(\beta) & = (\mathbf{y} - \mathbf{X}\beta)^T(\mathbf{y} - \mathbf{X}\beta) \\[8pt]
& = ((\mathbf y - P\mathbf y)+(P\mathbf y - X\beta))^T((\mathbf y - P\mathbf y)+(P\mathbf y - X\beta)) \\[8pt]
& = (\mathbf y-P\mathbf y)^T(\mathbf y-P\mathbf y) + (P\mathbf y-X\beta)^T(P\mathbf y-X\beta) \\
&\phantom{{}={}} + \underbrace{(\mathbf y-P\mathbf y)^T(P\mathbf y-X\beta) + (P\mathbf y-X\beta)^T(\mathbf y - P\mathbf y)}_{\text{This sum is zero, by orthogonality.}} \\[10pt]
& = (\mathbf y-P\mathbf y)^T(\mathbf y-P\mathbf y) + (P\mathbf y-X\beta)^T(P\mathbf y-X\beta).
\end{align}
Now $\beta$ appears only in the second term.  The second term can be minimized by making it $0$ since $P\mathbb y$ is in the column space of $X$ and therefore $\beta$ can be so chosen as to make $X\beta=P\mathbf y$.
The matrix $X$ has typically far more rows than columns.  If the columns of $X$ are linearly independent, then $X$ has an left-inverse, which is $(X^TX)^{-1}X^T$.  Invertibility of $X^TX$ follows from linear independence of the columns of $X$.
Hence the equation $X\beta=P\mathbf y$ can be solved for $\beta$ by multiplying both sides on the left by that left-inverse.
