1. A method for generating a label-based descriptor representing a labeled image, the method comprising: receiving an input image having a plurality of pixels and a plurality of labels, each label associated with one of the plurality of pixels, and each label indicating a class of objects depicted by the associated pixel from a predefined set of classes; generating, by one or more processors, a positional vector for the input image based on the plurality of labels, the positional vector representing positional information pertaining to each of the classes of objects depicted in the input image; generating, by one or more processors, an appearance vector for the input image based on the plurality of labels, the appearance vector representing appearance of the classes of objects depicted in the input image; generating, by one or more processors, a shape vector for the input image based on the plurality of labels, the shape vector representing a distribution of boundaries between pairs of the classes of objects, wherein generating the shape vector comprises evaluating an orientation distribution of boundaries between pairs of different classes of objects; and generating, by one or more processors, the label-based descriptor for the input image as a weighted combination of at least the positional vector, the appearance vector and the shape vector.