How to check convexity? How can I know the function $$f(x,y)=\frac{y^2}{xy+1}$$ with $x>0$,$y>0$ is convex or not?

The book "Convex Optimization" by Boyd, available free online here, describes methods to check.
The standard definition is if f(θx + (1 − θ)y) ≤ θf(x) + (1 − θ)f(y) for 0≤θ≤1 and the domain of x,y is also convex.
So if you could prove that for your function, you would know it's convex. 
The Hessian being positive semi-definite, as mentioned in comments, would also show that the function is convex.
See page 67 of the book for more.
