If $y = W^T x$, what is $\frac{\partial y}{\partial W}$? I would like to derive the derivative of a vector by matrix, i.e. $y = W^Tx$, where $W$ is a matrix, $x,y$ are vectors. What is $\frac{\partial y}{\partial W} = \frac{\partial W^T x}{\partial W}$?

Follow-up:
Define another function $z = a^T y = a^T W^Tx$, so that $z$ is a scalar. We know that $\frac{\partial z}{\partial W}$ is a matrix with the same of $W$. At the same time, by chain rule
\begin{equation}
\frac{\partial z}{\partial W} = \frac{\partial z}{\partial y} \cdot \frac{\partial y}{\partial W} 
\end{equation}
where $\frac{\partial z}{\partial y}$ is a $1\times N$ vector ($N$ is the dimension of $y$). So it seems the dimensions of matrices in left and right hand side don't match. Any explanations?

Consider the full matrix version of this problem, written in terms of the Frobenius (:) Inner Product 
$$\eqalign{
 Y & = W^TX \cr
 z &= A:Y = XA^T:W  \cr
}$$
The gradient of $z$ can be evaluated directly
$$\eqalign{
 dz &= XA^T:dW \cr
\frac{\partial z}{\partial W} &= XA^T \cr\cr
}$$
It can also be evaluated by the chain rule
$$\eqalign{
 dz &= A:dY \cr
\frac{\partial z}{\partial Y} &= A \cr\cr
 dY &= dW^TX &= {\mathbb E}X^T:{\mathbb B}:dW \cr
\frac{\partial Y}{\partial W} &= {\mathbb E}X^T:{\mathbb B} \cr
}$$
Taking the inner product of these 2 gradients (note that the second one is a 4th order tensor) yields
$$\eqalign{
\frac{\partial z}{\partial W} &= \frac{\partial z}{\partial Y} :
\frac{\partial Y}{\partial W} \cr
 &= A:{\mathbb E}X^T:{\mathbb B} \cr
 &= AX^T:{\mathbb B} \cr
 &= XA^T \cr
}$$
The 4th order isotropic tensors have components
$$\eqalign{
 {\mathbb E}_{ijkl} &= \delta_{ik}\,\delta_{jl} \cr
 {\mathbb B}_{ijkl} &= \delta_{il}\,\delta_{jk} \cr
}$$
and have the interesting properties when multiplied by a matrix $M$, using the Frobenius product
$$\eqalign{
 {\mathbb E}:M = M:{\mathbb E} &= M \cr
 {\mathbb B}:M = M:{\mathbb B} &= M^T \cr
}$$
Having derived this for the full matrix case, the result also holds when the matrices $(Y, X)$ are replaced by the vectors $(y, x)$.
