Get covariance of pixels in an image? What is the best way to calculate the covariance matrix for a small square of pixels? Assume it is a $3\times 3$ square of grayscale values.
I have read that in some cases, you can get the covariance matrix by inverting the Hessian. So, I was thinking of using a gradient operator to estimate the second derivatives ($J_{xx}, J_{yy}, J_{xy}$). Then, I believe the covariance would be this:
$$
\frac1{J_{xx}J_{yy} - J_{xy}^2}
\begin{bmatrix}J_{yy} & -J_{xy} \\-J_{xy} & J_{xx}\end{bmatrix}
$$
Again, I'm not sure if that's correct. Alternatively, I was considering just trying to apply the covariance formula directly. I believe to do that, I'd have each pixel's center $(x,y)$ coordinate be a separate sample. Then calculate covariance, weighting each sample by the pixel color.

From my tests, it appears that inverting the Hessian does not give the correct covariance matrix. I'm not sure under what circumstances the inverted Hessian will give the correct answer.
What I ended up doing is assigning each pixel becomes a vector $x$ based on its pixel coordinates:
$$
\begin{matrix}
(-1,-1)&(0,-1)&(1,-1)\\
(-1,0)&(0,0)&(1,0)\\
(-1,1)&(0,1)&(1,1)
\end{matrix}
$$
Then, using the color of each pixel as the weight $w$, you can compute the matrix with (see wikipedia):
$$
n = \sum_{i=0}^8 w_i\\
m = \sum_{i=0}^8 w_i^2\\
\overline x = \frac1n\sum_{i=0}^8 w_ix_i\\
Cov = \frac n{n^2-m}\sum_{i=0}^8 w_i(x_i-\overline x)^T(x_i-\overline x)
$$
