1. A computer-implemented method comprising: displaying, by a computer system, an image having a region to which content is to be provided in generating a modified image and an outside region, wherein the region has a first sub-region and a second sub-region, wherein the outside region at least partially surrounds the region; receiving an input comprising a semantic mark to be placed on the image, the semantic mark indicating the first sub-region and the outside region; determining, by the computer system applying a patch-based optimization algorithm to the image, first content to be added to the first sub-region and second content to be added to the second sub-region adjacent to the first sub-region, wherein determining the first content and the second content comprises: restricting, based on the semantic mark indicating the first sub-region and the outside region, the patch-based optimization algorithm to using the outside region for determining the first content such that an area of the image other than the outside region is not used to identify the first content, identifying, using the patch-based optimization algorithm, the first content based on the outside region to which the patch-based optimization algorithm is restricted and not based on the area of the image that the semantic mark does not indicate, and identifying the second content for the second sub-region without being restricted to the outside region; and generating the modified image by modifying the image to include the identified first content in the first sub-region and the identified second content in the second sub-region.