1. A non-transitory computer readable medium embodying a program of instructions executable by machine to perform operations for detecting anatomy orientation, the operations comprising: (i) receiving an image volume of at least a portion of a structure of interest, (ii) extracting appearance features from the image volume, (iii) applying a pre-trained regressor to the appearance features of the image volume to predict a colatitude of the structure of interest, (iv) determining an optimal longitude of the structure of interest corresponding to the predicted colatitude, including uniformly sampling a set of longitudes at the predicted colatitude, re-orienting the image volume to each of the sampled longitudes to generate re-oriented image volumes, applying the pre-trained regressor to each of the re-oriented image volumes to predict relative colatitudes, and selecting one of the sampled longitudes with a smallest relative colatitude as the optimal longitude, (v) in response to the predicted colatitude being more than a pre-determined threshold, re-orienting the image volume based on the predicted colatitude and the optimal longitude, and repeating operations (ii), (iii) and (iv), and (vi) outputting the predicted colatitude and the optimal longitude.