Matrix Derivative Product Rule $\frac{d}{dx} \langle x,y\rangle Ax $ Let $A\in\mathbb{R^{n\times n}}$,$x,y\in\mathbb{R^{n}}$. What is the derivative of the following expression?
$$
\frac{d}{dx} \langle x,y\rangle Ax 
$$
What I tried:
Applying the product rule doesn't work because the first term (see next equation) is inconsistent dimensionally, or I am applying it in the wrong way. Let $f(x) = \langle x,y \rangle, g(x) = Ax$, then
$$ \frac{d}{dx} f(x)g(x) = \frac{df(x)}{dx} g(x) +  f(x)\frac{dg(x)}{dx} = xAx + \langle x,y \rangle A$$.
Here the first term in the last equation doesn't make sense dimensionally, any help or suggestion would be appreciated.

Define the vector-valued function
$$\eqalign{
f &= (y^Tx)(Ax) \\
}$$
and calculate its differential and gradient
$$\eqalign{
df &= (Ax)(y^Tdx) + (y^Tx)(A\,dx) \\
   &= \Big(Axy^T + (y^Tx)A\Big)\,dx \\
   &= A\Big(xy^T + (y^Tx)I\Big)\,dx \\
\frac{\partial f}{\partial x} &= A\Big(xy^T + (y^Tx)I\Big) \\
}$$
which is another way to derive JG's result while avoiding index notation.
Your error was to assume that the product rule is valid for the gradient of a vector-valued function like $f$, but it isn't.
The problem is that a gradient operation changes the tensorial character of any term to which it is applied (i.e. it turns vectors into matrices), and matrix multiplication is inherently non-commutative. Index notation was developed to handle precisely such calculations, but with standard matrix notation you must proceed very cautiously.
