Matrix/Vector Derivative I am trying to compute the derivative:$$\frac{d}{d\boldsymbol{\mu}}\left( (\mathbf{x} - \boldsymbol{\mu})^\top\boldsymbol{\Sigma} (\mathbf{x} - \boldsymbol{\mu})\right)$$where the size of all vectors ($\mathbf{x},\boldsymbol{\mu}$) is $n\times 1$ and the size of the matrix ($\boldsymbol{\Sigma}$) is $n\times n$.
I tried to break this down as $$\frac{d}{d\boldsymbol{\mu}}\left( \mathbf{x}^\top\boldsymbol{\Sigma} \mathbf{x} - \mathbf{x}^\top\boldsymbol{\Sigma} \boldsymbol{\mu} - \boldsymbol{\mu}^\top\boldsymbol{\Sigma} \mathbf{x} + \boldsymbol{\mu}^\top\boldsymbol{\Sigma} \boldsymbol{\mu} \right) $$
yielding $$(\mathbf{x} + \boldsymbol{\mu})^\top\boldsymbol{\Sigma} + \boldsymbol{\Sigma}(\boldsymbol{\mu} - \mathbf{x})$$
but the dimensions don't work: $1\times n + n\times 1$. Any help would be greatly appreciated.
-C

Think of when you compute the derivative with $n = 1$ : you get $(x-\mu)^{\top} \Sigma (x-\mu) = \sigma (x-\mu)^2$ for some constant $\sigma$ representing the matrix, thus derivative with respect to $\mu$ gives $2 \sigma(\mu-x)$. This is what happens in dimension $n$ : the derivative of this function is the gradient seen as a function of $\mu$. Write $x = (x_1, \dots, x_n)$, $\mu = (\mu_1, \dots, \mu_n)$ and $\Sigma = (\sigma_{ij})$. Then
$$
f(\mu) = (x-\mu)^{\top} \Sigma (x-\mu) = \sum_{i=1}^n \sum_{j=1}^n (x_i - \mu_i)(x_j - \mu_j)\sigma_{ij}.
$$
Computing partial derivatives, say, with respect to the $k^{\text{th}}$ variable, with $1 \le k \le n$, you get
$$\begin{align*}
\frac{\partial f}{\partial\mu_{k}} &= \sum_{i=1}^n \sum_{j=1}^n \left( -\delta_{ik} (x_j - \mu_j) \sigma_{ij} \right) + \left( (x_i -\mu_i) (-\delta_{jk}) \sigma_{ij} \right)\\
&= \sum_{j=1}^n (\mu_j - x_j) \sigma_{kj} + \sum_{i=1}^n (\mu_i - x_i) \sigma_{ik},
\end{align*}$$
where $\delta_{ij} = 0$ if $i \neq j$ and $1$ if $i=j$. If you look at the vector $\nabla f = \left( \frac{ \partial f}{\partial \mu_1} , \dots, \frac{ \partial f}{\partial \mu_n}\right)$, you see that its components are precisely those of the vector $\Sigma(\mu-x) +  \Sigma^{\top} (\mu-x)$. If the matrix $A$ is symmetric, you get $2\Sigma(\mu-x)$.
Hope that helps,
