Kalman decomposition using Hautus test In linear control theory, the Kalman decomposition is used as a similarity transformation to decompose a given linear time-invariant system 

$$\dot{x}(t)=Ax(t)+Bu(t)$$
$$y(t) = Cx(t) + Du(t)$$

into its controllable &amp; observable, controllable &amp; not observable, not controllable &amp; observable and not controllable &amp; not observable subsystems. 

A system is controllable if $\mathcal{C}=[B\quad AB \quad A^2B \ldots A^{n-1}B]^T$ has rank $n$ (number of rows/columns of the system matrix $A$).

A system is observable if $\mathcal{O}=[C \quad CA \quad CA^2 \ldots CA^{n-1}]^T$ has rank $n$ (number of rows/columns of the system matrix $A$). 

It seems possible to construct the similarity transformation by using the eigenvectors of the Hautus test for controllability and observability.

It states a system is controllable if for all eigenvalues $\lambda$ of $A$ the matrix $[A-\lambda I\quad B]$ has full rank $n$ for a $n \times n$ system matrix $A$. A similar statement reads: A system is observable if for all eigenvalues $\lambda$ of $A$ the matrix $[A^T - \lambda I \quad C^T]$ has full rank $n$ for a $n \times n$ system matrix $A$.

The user fibonatic from Engineering Stack Exchange suggested that one could construct the similarity transformation for the Hautus test. I am wondering, how this is possible.

This is the same example as in the post on Engineering Stack Exchange.

$$\dot{x} = \begin{bmatrix}1 &amp; 1 &amp; 0 \\ 0 &amp; 1 &amp; 0 \\ 0 &amp; 1 &amp; 1 \end{bmatrix}x + \begin{bmatrix}0 &amp; 1 \\ 1 &amp; 0 \\ 0 &amp; 1 \end{bmatrix} u$$
$$y = \begin{bmatrix}1 &amp; 1 &amp; 1 \end{bmatrix} x$$

The similarity transfromation is given by 
$$
M = 
\begin{bmatrix}
0 &amp; 1 &amp; 1 \\
1 &amp; 0 &amp; 0 \\
0 &amp; 1 &amp; -1
\end{bmatrix}.
$$
This is what I have tried. The three eigenvalues of $A$ are all $\lambda =1$ the eigenvectors are $v_1 = [1, 0,0]^T, v_2=[0,0,1]^T$ and $v_3=[1,0,1]^T$. The rank of the $[A-\lambda B]$ is 2. That means there is a deficit in rank for every eigenvector. If I include the eigenvectors to a "similarity" transformation

$$\tilde{M} =
\begin{bmatrix}
1 &amp; 0 &amp; 1\\
0 &amp; 0 &amp; 0\\
0 &amp; 1 &amp; 1\\ 
\end{bmatrix}
$$ 

it turns out that it is singular and cannot be the similarity transformation that I need. 


  So how can I use the Hautus test to construct the similarity
  transformation for the Kalman decomposition?
A.Γ. is right that you need to use the generalised eigenvectors, so do a Jordan decomposition instead of an eigendecomposition. So in case of the example in your question the $A$ matrix has the following generalised eigenvectors

$$
V = 
\begin{bmatrix}
1 &amp; 0 &amp; 0 \\
0 &amp; 1 &amp; 0 \\
1 &amp; 0 &amp; 1
\end{bmatrix}
$$

where the Jordan canonical form of $A$ consists out of two blocks of size two and one, both with the eigenvalue one. 

However $\text{rank}\left[A-\lambda\,I \quad B\right]$ and $\text{rank}\left[A^\top-\lambda\,I \quad C^\top\right]$ tells you nothing about which block and corresponding generalized eigenvectors are causing those matrices to lose rank. So this would be the limitation of using the Hautus test in order to obtain the Kalman decomposition. The structure of a Jordan block actually always ensures that the largest rank loss caused by that block is at most one. So all this method would tell you is how many blocks have rank loss, but not which and by how much. For example when

$$
A = 
\begin{bmatrix}
0 &amp; 1 &amp; 0 \\
0 &amp; 0 &amp; 1 \\
0 &amp; 0 &amp; 0
\end{bmatrix}, \quad
B_1 = \begin{bmatrix}
0 \\ 1 \\ 0
\end{bmatrix}, \quad
B_2 = \begin{bmatrix}
1 \\ 0 \\ 0
\end{bmatrix}
$$

where $A$ is one Jordan block of size three with eigenvalue zero, but for both $B_1$ and $B_2$ does $\left[A-\lambda\,I \quad B_i\right]$ lose rank. But for $B_1$ only one state is uncontrollable, while for $B_2$ two states are uncontrollable.

So this method fails when the matrix has an eigenvalue with algebraic multiplicity larger than one, but the rank loss (either for controllability, observability or both) is less then that. Because in that case it is not clear which subset of (generalised) eigenvectors needs to be taken.
Eigenvector $v_3=v_1+v_2$ which implies that your eigenvectors aren't independent.