How is this case a reflexive relation? My textbook says, if $R$ is a reflexive relation on set $A=\{1,2,3,4\}$ then $R = \{ (x,x) : \forall x \in A\}$ then it proceeds on with $2$ examples.

$$R_1= \{ (1,1),(2,2),(3,3),(4,4) \}$$ Alrighty, this is fine and makes sense. Every $x\in A$ there is an $(x,x)$ type of elements in $R_1$

Now the trouble starts with this example,

$$R_2= \{(1,1),(2,2),(3,3),(4,4),(1,2) \} $$ is also a reflexive relation?

My question : 


  In $R_2$ there exists an element $(1,2)$ which is not of the type $(x,x)$ but we clearly said earlier the Relation will contain all $(x,x)$ type elements. Then why is this a reflexive relation?
The relation can contain other elements too. It must contain all the elements of the type $(x,x)$ in order to be reflexive but it doesn't mean that these are the only elements in the relation.
Your $R_1$ is an example of an identity relation. Which is indeed also a reflexive one. But not every reflexive relation is identity. 

Reflexive relation has its elements of the $\{x, x\}$ type where $x$ belongs to $A$. When all identity types are contained in the set then the reflexive relation can contain more than those too. 
Therefore:
$$R=\{[(x, x) :x \text{ belongs to $A$ }] \text{ or } [\{x, y\} :x \text{ belongs to $A$, $y$ belongs to $A$}]\} $$
This can be the complete definition for reflexive set.