stability and asymptotic stability: unstable but asymptotically convergent solution of nonlinear system Consider nonlinear systems of the form $X(t)'=F(X(t))$, where $F$ is smooth (assume $C^\infty$). Is it possible to construct such a system (preferably planar system) so that $X_0$ is an unstable equilibrium, but all nearby solution curves tend to $X_0$ as $t \to \infty$? If so, how? A conceptual construction is enough. What will the phase portrait look like?


Definition (Verhulst 1996)
The equilibrium solution $X_c$ is called asymptotically stable if there exists a $\delta(t_0)$ such that
\begin{equation}
||X_0 - X_c|| \le \delta(t_0) \implies \lim_{t\rightarrow\infty} ||X(t;t_0,X_0) - X_c|| =0
\end{equation}

So no, you're essentially asking can a system be both asymptotically stable and unstable which is a contradiction.
EDIT:
Glendinning's text refers to this definition as quasi-asymptotic stability to differentiate it from 'direct' asymptotic stability (solutions are both quasi-asymptotically stable and Lyupanov stable).
As an example he presents this equation:
\begin{equation}
\dot{r} = r(1-r^2) \qquad \dot{\theta} = 2\sin^2\left(\frac{1}{2}\theta\right),
\end{equation}
which has an unstable critical point at $(0,0)$ a quasi-asymptotically stable critical point at $(1,0)$.
Points are attracted to the invariant circle at $r=1$. On this curve, the flow is semi-stable. If $\pi>\theta >0$ then the flow wraps around until it passes $\theta = \pi$. If $ -\pi> \theta>0$ the trajectory falls into the critical point at $(1,0)$. This is an example of a homoclinic orbit.
In the example provided, the critical point at $(1,0)$ is a saddle. Which, if all we considered was the linearised system, is unstable. Indeed it is Lyapunov unstable as there is a trajectory that leaves any $\epsilon$-ball about $(1,0)$.
However, in nonlinear systems the linearisation is often not sufficient,
especially with saddles, which are often the cause of a whole bunch of interesting behavior that aren't represented in the linearisation, like homoclinic and heteroclinic connections.
I think the confusion comes from how you/they define instability.
The texts I refer to define a critical point as unstable if it is not Lyapunov stable or 'Quasi-Asympotically' stable. However if all you go off is the linearisation then i'd imagine the example is what you're after.
