1. A method of performing motion compensation, comprising: dividing a current frame of video data into blocks; and clustering the blocks based on motion vectors, wherein clustering the blocks comprises: determining a motion pair for a current block, wherein the motion pair consists of a foreground and background motion vector for each block; if the current block is other than a first block used, finding a distance from the motion pair from centers of any current clusters; and clustering the current block into a cluster based upon the distance.