$F(F(x)+x)^k)=(F(x)+x)^2-x$ I have no idea about this problem. But I feel we have to use chain rule of differentiation here.
The Function $F(x)$ is defined by the following identity:
$F(F(x)+x)^k)=(F(x)+x)^2-x$
The value of $F(1)$ is such that a finite number of possible values of $F'(1)$ can be determined solely from the above information.The maximum value of $k$ such that  $F'(1)$ is an integer can be expressed as $\frac{a}{b}$, where $a , b$ are co-prime integers. What is the value of $a+b$

$$F((F(x)+x)^k) = (F(x)+x)^2 - x
$$
differentiate both sides to get
$$F'((F(x)+x)^k)\cdot k (F(x)+x)^{k-1}\cdot (F'(x)+1) = 2(F(x)+x)\cdot(F'(x)+1) -1
$$
Let denote $u(x)=F(x)+x$ then
$$ F'(u(x)^k) \cdot k u(x)^{k-1} u'(x) = 2u(x)u'(x) -1
$$
The only appropriate value for $F(1)$ I could think is $F(1)=0$, i.e. $u(1)=1$ , then 
$$ F'(1) k  u'(1) = 2u'(1) -1$$ 
that is
$$ k F'(1) (F'(1)+1)=k (F'(1)^2 +F'(1)) = 2(F'(1)+1) -1 = 2F'(1)+1$$
or
$$ k F'(1)^2 + (k - 2) F'(1)-1 = 0$$
Thus, we got finitely many solutions for $F'(1)$ as required:
$$ F'(1)_{\pm}=\frac{2-k \pm\sqrt{k^2+4}}{2 k}$$
Wrong solution
One takes limit of this expression as $k\to\infty$ and get $-1$ for $F'(1)_-$ ( and $0$ for $F'(1)_+$ which we'll drop). Thus, we got $a=-b$ and therefore $a+b=0$. 
Correct Solution
(thanks to @A.S. whose comment triggered me to finish it)
The requirement is that $F'(1)$ be integer, let $n$ be that integer, i.e.
$$\frac{2-k \pm\sqrt{k^2+4}}{2 k}=n$$
$$(2kn-2+k)^2=k^2+4$$
$$(2kn-2+k)^2-k^2=4$$
$$((2kn-2+k)-k)((2kn-2+k)+k)=4$$
$$4(kn-1)(kn-1+k)=4$$
$$(kn-1)(k(n+1)-1)=1$$
Solve it for $k\ne 0$ (since $0$ is not the larges solution) to get $$k=\frac{2 n+1}{n (n+1)}.$$ 
Obviously, the maximum $k$ for an integer $n$ (in the domain of definition) is achieved at $n=1$, thus $$k=\frac{a}b=\frac{3}2$$ and therefore $$a+b=5$$
