1. A method comprising: identifying a training set of images, wherein each image in the training set has an identified bounding box that comprises an object class and an object location for an object in the image; segmenting each image of the training set, wherein segments comprise sets of pixels that share visual characteristics, and wherein each segment is associated with an object class; clustering the segments based on object classes; and generating, by a processing device, a data structure associated with the training set of images based on the clustering, wherein each of a plurality of entries in the data structure comprises a set of visual characteristics for a prototypical segment from the training set of images and further comprises an associated set of potential bounding boxes, the prototypical segment representing a combination of similar segments with objects of a same object class across different images from the training set of images, wherein the data structure is usable to predict bounding boxes of additional images that each include an object having an object class.