1. A computer-implemented method for tracking an object in a frame of a video, comprising: (a) determining an adaptive term value based on an adaptive model and at least a portion of the frame, wherein the adaptive model is determined based on at least one previous frame of the video; (b) determining a pose constraint value based on a pose model and at least a portion the frame, wherein the pose model is determined based on different possible appearances of the object; (c) determining an alignment confidence score based on an alignment model and at least a portion the frame, wherein the alignment model detects misalignments of the tracked object; (d) determining an energy value based on the adaptive term value, the pose constraint value, and the alignment confidence score; and (e) determining a resultant tracking state based on the energy value, the resultant tracking state defining a likely position of the object in the frame given the object's likely position in a set of previous frames in the video.