1. A computer-implemented method for propagating curves onto new key frames in a sequence of frames of video data, comprising: receiving one or more curves drawn on a first key frame to define a boundary of an object in the first key frame, the one or more curves including multiple endpoints and control points; mapping the one or more curves to a second key frame that includes the object, the mapping performed without interpolation of the one or more curves between the first key frame and an end boundary in the second key frame; generating a confidence value for respective subregions of a plurality of subregions of the second key frame that include at least a portion of the one or more curves that are mapped to the second key frame, the confidence value representing a measure of separability of foreground and background colors for each respective subregion; generating a confidence map of the confidence values to indicate a level of confidence for a predicted position of each control point in the second key frame, the confidence map including a blend of the confidence values of the respective subregions that are along a new or changed boundary of the object in the second key frame; and updating the multiple endpoints and control points of the one or more curves to fit the new or changed boundary of the object in the second key frame, the updating being based on the confidence map and the confidence value associated with the respective subregions that include one or more of the control points.