1. A system comprising: a non-transitory computer-readable medium storing a data structure and image patches; a processing device communicatively coupled to the non-transitory computer-readable medium, the processing device configured to access the data structure and the image patches stored in the non-transitory computer-readable medium, wherein the processing device is further configured to execute program code that, when executed by the processing device, configures the processing device to perform operations comprising: identifying an offset between a first source image patch and a first target image patch; adding, to the data structure, an entry with a label specifying the identified offset; associating, via the data structure, the first source image patch with the label, wherein the identified offset is included in a set of high-frequency offsets stored in the data structure; retrieving, from the data structure, (i) first frequency data indicating that the identified offset is associated with a first number of source image patches and (ii) second frequency data indicating that an additional offset is associated with a second number of source image patches; selecting, from the set of high-frequency offsets, the identified offset rather than the additional offset based on the first number of source image patches being larger than the second number of source image patches; identifying, using the identified offset selected from the set of high-frequency offsets, a second target image patch that is located at the identified offset from a second source image patch; determining that the second source image patch is sufficiently similar to the second target image patch; and based on the second source image patch and the second target image patch being sufficiently similar, associating, via the data structure, the second source image patch with the entry for the identified offset.