1. A computer-implemented method of detecting object boundaries in a video, comprising: identifying a pixel of an image frame of a video; defining a patch for the pixel; identifying a set of neighboring image frames for the image frame; detecting a set of patch matches to the patch from the set of neighboring image frames; and calculating an object boundary probability value for the patch corresponding to the pixel by applying a difference metric to the patch and the set of patch matches.