Covariance of states of a finite Markov chain I know it is possible to construct a covariance matrix for states of a Markov chain but I cannot seem to find a proper way to compute it. I will attach some theories I found from Kemeny and Snell's FMCs which states about limiting covariance.







I just cannot understand how the covariance matrix can be generated using this. I need to know whether there is a much easier way to approach this problem. In the same text in an example the have computed the convariance matrix without any steps showing how it was computed. 


Hope someone could help me out on this and give me a better insight to the concept of covarince with respect to states in a Markov chain. Thanks !
Try this (R code) 
   a &lt;- A[1,]   # Limiting matrix
   C &lt;- mat.or.vec(m,m)  # mxm covariance matrix
   delta &lt;- function(i,j)  {delta &lt;- 0; if (i==j) delta &lt;- 1; return(delta)}
   for (i in 1:m)   {
      for (j in 1:m)   {
         C[i,j] &lt;- a[i]*Z[i,j]+a[j]*Z[j,i] - a[i]*a[j] - a[i]*delta(i,j)
      }
   }
Result:


  375*C
       [,1] [,2] [,3]
  [1,]  134  -18 -116
  [2,]  -18   36  -18
  [3,] -116  -18  134
  Kemeny's example on page 89.
Try this (R code) 
   a &lt;- A[1,]   # Limiting matrix
   C &lt;- mat.or.vec(m,m)  # mxm covariance matrix
   delta &lt;- function(i,j)  {delta &lt;- 0; if (i==j) delta &lt;- 1; return(delta)}
   for (i in 1:m)   {
      for (j in 1:m)   {
         C[i,j] &lt;- a[i]*Z[i,j]+a[j]*Z[j,i] - a[i]*a[j] - a[i]*delta(i,j)
      }
   }
Result:


  375*C
       [,1] [,2] [,3]
  [1,]  134  -18 -116
  [2,]  -18   36  -18
  [3,] -116  -18  134
  Kemeny's example on page 89.