How to find a counter example for non convexity? Consider a simple function $f(x,y)=\frac{x}{y}, x,y \in (0,1]$, the Hessian is not positive semi definite and hence it is a non convex function. However, when we plot the function using Matlab/Maxima, it "appears" convex. For the sake of clarity we want to find points which violate the definition of convexity, in other words identify the region (in the plot) where the function is non-convex. Can you suggest a procedure to do the same ?        

If you write down the Hessian matrix you'll see that for $x,y\in (0,1]$ the Hessian is indeed positive semi-definite. This is call conditional positive semi-definiteness. You can see the Hessian as a matrix parametrized by $x$ and $y$: it is positive semi-definite depending on their values. Try to compute the eigen-values as a function of $x$ and $y$.
