1. A method for classifying an image, comprising: inferring location information of an object of interest in an input representation of the image; determining foreground object features and background object features from the input representation of the image; pooling the foreground object features separately from the background object features using the location information to form a new representation of the image, the new representation being different than the input representation of the image; classifying the image based on the new representation of the image; performing an outer iterative loop, the outer iterative loop comprising: initializing a background region from the input representation of the image; and incrementally shrinking at least a smallest bounding box from among a plurality of varying sized bounding boxes applied to the background region to incrementally increase a size of the background region with respect to at least the smallest bounding box; performing an inner iterative loop that cooperates with the outer iterative loop, the inner iterative loop comprising: inferring the location information of the object of interest from a current version of the smallest bounding box; and from among a set of positive bounding boxes from positive images that are known to include the object of interest therein and from among a set of negative bounding boxes from negative images that are known to omit the object of interest therein, training a support vector machine classifier to discriminate the positive bounding boxes from the negative bounding boxes to refine the location information of the object of interest, wherein said classifying step classifies the image based on discrimination results applied to a binary label of the image.