Showing system contains peroidic orbit Consider the IVP for the $2$-dimensional dynamical system ($X=[0, \infty )^2$) $$\dot{x_1}=a-x_1-\frac{4x_1x_2}{1+x_1^2}$$ $$\dot{x_2}=bx_1 \bigg( 1- \frac{x_2}{1+x_1^2} \bigg)$$ for all $t \in I$, and $\textbf x (0)=(x_{10}, x_{20}) \in [0, \infty )^2$, with $a,b &gt;0$.

Show that for certain $a$, $b$, and $c$, there exists an invariant region
$$\{f(x_1, x_2) \in X : 0 &lt; x_1 &lt; a, \, \, 0 &lt; x_2 &lt; c \}$$
that contains a periodic orbit for the dynamical system.

My attempt: 

So we have a rectangle and we have to choose $a,b,c$ such that the flow always remains within the rectangle. Let's consider each edge and corner of the rectangle. Imagine our graph to be $x_1$ as the horizontal axis and $x_2$ as the vertical.

At corner $(0,0)$: $\dot x_1=a$, $\dot x_2 =0$ so at this point, flow is directed $x_1$ direction so this is inside the region.

At corner $(0,c)$: $\dot x_1=a$, $\dot x_2 =0$ so at this point, flow is directed $x_1$ direction so this is inside the region.

At line $x_1=0$ so on points $(0,l)$ with $l \in (0,c)$: $\dot x_1=a$, $\dot x_2 =0$ so at this point, flow is directed $x_1$ direction so this is inside the region.

At corner $(a,c)$: $$\dot x_1 =-\frac{4ac}{1+a^2}&lt;0, \, \, \forall a,c$$

$$\dot x_2 =ba \bigg( 1- \frac{c}{1+a^2} \bigg)$$ with $ba&gt;0, \, \, \forall a,b$ but we need the sign of $\dot x_2$ to be $\leq 0$ and this is dependent on the sign of the following (so we WANT the following): $$ 1- \frac{c}{1+a^2} \leq 0$$ so then the flow at this corner will go negative $x_1$ direction or down within the region. Either way, flow will still be in the region.

At line $x_2=0$ so on points $(q,0)$ with $q \in (0,a)$: $\dot x_1 =a-q &gt;0, \, \, \forall a,q$, and $\dot x_2 =bq&gt;0$. So the flow is directed in the region.

At line $x_1=a$ so on points $(a,p)$ with $p \in (0,c)$: $$\dot x_1 = - \frac{4ap}{1+a^2}&lt;0, \, \, \forall a,p$$ 
$$\dot x_2 =ba \bigg( 1- \frac{p}{1+a^2} \bigg)$$ We are unsure on the sign of this but regardless the flow will remain in the region since $\dot x_1 &lt;0$.

At line $\dot x_2 =c$, so on points $(t,c)$ with $t \in (0,a)$: $$\dot x_1 = a-t -\frac{4tc}{1+t^2}&lt;0 $$

$$\dot x_2 =bt \bigg( 1- \frac{c}{1+t^2} \bigg)$$ We are unsure on the signs of both of these but we WANT $\dot x_2 \leq 0$ and then regardless of the sign of $\dot x_1$, the flow will remain in the region.

So we need the flowing inequalities to hold: $$1- \frac{c}{1+a^2} \leq 0$$ $$1- \frac{c}{1+t^2} \leq 0$$ Choose $c=66$ and $a=8$ and then they hold. 

Next we need our fixed point to be out of the region and this happens when $$1 + \frac{a^2}{25} &lt; c$$ and this is true for our chosen values of $a$ and $c$. Hence by Poincare-Bendixon theorem, there exists a periodic orbit.

IS THIS CORRECT? PLEASE COMMENT!!

I think we have to use this:

Poincare-Bendixon: Consider $\dot x = f(x)$ with $f \in C^1(X)$ (continuous on first differential) and $X \subset \mathbb R^2$. If the forward orbit, $O^+(x_0)$ (for $x_0 &gt; X$) is
contained within a closed bounded set $D$ containing no fixed points, then
$\omega (x_0)$ contains a periodic orbit.

Generalized Poincare-Bendixon: Let the hypothesis of the above hold, except now, let $D$ contain fixed
points $p_1, ..., p_N$. Then $\omega (x_0)$ is one of the following;

1) a fixed point $p_i$.

2) a periodic orbit.

3) a finite number of fixed points $p_i$ , and a countable number of
homoclinic/ heteroclinic orbits between $p_i$ and $p_j$.
Hint:

Find the directions of the vector field along the sides of the rectangle $[0,a]\times[0,c]$, for some carefully chosen $c$. For example, along the side $x_1=0$ you have $\dot x_1=a$ and $\dot x_2=0$, showing that the vector field points inside the rectangle. Similarly, for $x_2=0$ you have $\dot x_1=a-x_1$ and $\dot x_2=bx_1$, and again the vector field points inside the rectangle (you see the pattern).

After that study the vector field in a neighborhood of the equilibrium point.

A minor comment about the wording: you should really write "forward invariant" instead of "invariant", and "equilibrium point" instead of "fixed point".
to make life easier, i am going to pick $a = 5, b = 1$ so that we have one rest point $x_1 = 1, x_2 = 2.$ we will make a change of variable 
$$\xi_1 = x_1 - 1, \xi_2 = x_2 - 2 \text{ and } x_1 = 1+\xi_1 , x_2 = 2+\xi_2.$$  the governing equations for $\xi_1, \xi_2$ are:

$$\dot \xi_1 = 5 - (1+\xi_1 ) - \frac{4(1+\xi_1)(2+\xi_2)}{1+(1+\xi_1)^2}\\
\dot \xi_2 = (1+\xi_1)\left( 1- \frac{2+\xi_2}{1+(1+\xi_1)^2}\right) $$ which simplifies to  $$\dot \xi_1 = \frac{-2\xi_1-4\xi_2+2\xi_1^2-\xi_1^3-4\xi_1\xi_2}{1+(1+\xi_1)^2}\\
\dot \xi_2 = \frac{(1+\xi_1)(2\xi_1-\xi_2+\xi_1^2)}{1+(1+\xi_1)^2}$$ 
the associated linear system has the coefficient matrix $\pmatrix{-1&amp;-2\\2&amp;-1}$ with the eigenvalues $-1\pm 2i$ suggesting that the fixed point is a stable spiral.

i will now now do transformation to polar coordinates $$\xi_1 = r\cos \theta, \xi_2 = r \sin \theta. $$

i will come back to this tomorrow if you don't get to it by then.