The system of three DE I want to solve the following system of DE:
$$
\begin{cases} \dot{x} = 2x+6y -15z,  \\ \dot{y} =x+y-5z,\\ \dot{z} = x+2y-6z, \end{cases}
$$
First, I rewtite the coefficents in the matrix form:
$$A = \begin{bmatrix}
2 & 6 & -15\\
1 & 1 & -5\\
1 & 2 & -6
\end{bmatrix}$$Then, I find $$det(A-\lambda I) = \begin{vmatrix}
2 -\lambda& 6 & -15\\
1 & 1 -\lambda& -5\\
1 & 2 & -6-\lambda
\end{vmatrix}
=-(\lambda+1)^3
$$
$\lambda=-1$ is of multiplicity $3$ and I don't know how to continue.

For $\lambda =-1$, the rank of $A- \lambda I $ is 1, so the dimension of the corresponding eigenspace is 2, which is not equal to the multiplity of the eigenvalue in its characteristic equation, which is 3. Thus the matrix is not diagonalizable. You need to learn about the Jordan form and generalized eigenvectors, which are solutions to $$(A- \lambda I)^nv=0$$ for $n>1.$ I'm not going to do the details for you, but using generalized eigenvectors you can construct a matrix $P$ such that $P^{-1}AP$ has -1 on the leading diagonal and either one  or two 1's on the super-diagonal. Thus exactly one of the following will be true: (i) there exists a non-singular $P$ such that $$P^{-1}AP=\begin{bmatrix}-1&1&0\\0&-1&1\\0&0&-1 \end{bmatrix}$$ or (ii) there exists a non-singular $P$ such that $$P^{-1}AP=\begin{bmatrix}-1&0&0\\0&-1&1\\0&0&-1\end{bmatrix}$$ In either case, let $$\begin{bmatrix}x\\y\\ z\end{bmatrix}=P\begin{bmatrix}\eta\\ \zeta \\ \xi\end{bmatrix}$$. Then $$\begin{bmatrix}\dot{\eta}\\ \dot{\zeta}\\ \dot{\xi}\end{bmatrix}=P^{-1}AP\begin{bmatrix}\eta\\\zeta\\\xi\end{bmatrix}$$ Then you solve for first $\xi$, then $\zeta$ and finally $\eta$. Finally you can recover $x,y \text { and }z.$
