How to check convexity? How can I know the function $$f(x,y)=\frac{y^2}{xy+1}$$ with $x>0$,$y>0$ is convex or not?

Consider $y=x$ then we have $\displaystyle g(x)=\frac{x^2}{x^2+1}=1-\frac 1{x^2+1}$
The second derivative of this is $g''(x)=\frac{2-6x^2}{(1+x^2)^3}$
and will change sign around $x=\frac 1{\sqrt{3}}$ so that $g$ is convex in $(0,\frac 1{\sqrt{3}})$ and concave in $(\frac 1{\sqrt{3}},\infty)$.
Your function is clearly not convex nor concave on $(\mathbb{R^{+*}})^2$ but you could search more restricted sets if needed...
Here is a picture (from below) of your function (convex near $y=0$ and concave when $y$ becomes larger at least in the x=y direction, in the x=-y direction it looks convex...) : 

