1. A computer-implemented method, comprising: under control of one or more computing systems comprising one or more processors, receiving a first image and a second image; creating an appearance model of a first foreground object and a first background of the first image based on contextual color patterns of the first image; creating an appearance likelihood model of a likely foreground and a likely background of the second image, the appearance likelihood model including a cumulative likelihood energy of the likely foreground; creating an edge profile model of the first image based on learned color contexts near edges in the first image; creating an edge profile likelihood model of the second image based on additional learned color contexts of edges in the second image, the edge profile likelihood model including a likely segmentation boundary; and segmenting the second image into a second foreground object and a second background by optimizing the likely segmentation boundary in the second image through minimizing energy related to the appearance model and the appearance likelihood model, and minimizing a likelihood energy component and a smoothness energy component that is associated with the edge profile model and the edge profile likelihood model using a min-cut/max-flow technique.