The vector $\mathbf{x}$ is the derivative of $\mathbf{Ax}$ with respect to what? Consider the linear equation
$$\mathbf{A}\mathbf{x}=\mathbf{b},$$
where


*

*$\mathbf{A}=\begin{bmatrix}\mathbf{a}_1 \\ \vdots \\ \mathbf{a}_n\end{bmatrix}$ is an $N \times K$ random matrix

*$\mathbf{x}=\begin{bmatrix}x_1 \\ \vdots \\ x_n\end{bmatrix}$ is a $K \times 1$ vector

*$\mathbf{b}=\begin{bmatrix}b_1 \\ \vdots \\ b_n\end{bmatrix}$ is an $N \times 1$ random vector


For each $K \times 1$ row $\mathbf{a}_n$ of the matrix $\mathbf{A}$, observe that
$$\frac{\partial \mathbf{a}_n \mathbf{x}}{\partial a_k}=\frac{\partial (a_{n1} x_1 + a_{n2} x_2+\cdots+a_{nK} x_K)}{\partial a_{nk}}=x_k$$
Is there some way to generalize this statement for all columns $k$ simultaneously? In other words,

The vector $\mathbf{x}$ is the derivative of the system of equations $\mathbf{Ax}$, taken with respect to what?


Taking a quick look at this list of identities, I see that $\mathbf{A}$ is the derivative of $\mathbf{A}\mathbf{x}$ with respect to $\mathbf{x}$:
$$\mathbf{A}=\frac{\partial \mathbf{Ax}}{\partial{\mathbf{x}}}$$
However, I don't see what derivative would give us $\mathbf{x}$ as an answer:
$$\mathbf{x}=\frac{\partial \mathbf{Ax}}{\partial (?)}$$
I'm not particularly good with the rules of matrix differentiation, so a proof would be especially helpful here. Thank you!

With the usual notations, derivating with respect to a vector increases the dimension by 1, and derivating wrt to a matrix increases the dimension by 2. For example, if $s$ is a scalar, $v$ a vector and $A$ a matrix, $\partial_A s$ is a matrix, $\partial_v s$ is a vector and $\partial_v v$ is a matrix.
In your case, you differentiate a vector with respect to ? to obtain a vector. So, with usual notations, necessarily ? is a scalar. And I don't think any scalar satisfies this.
