1. A method for identifying scenes in captured video for inclusion in a video summary, the method comprising: accessing a video, the video including a plurality of frames; determining a velocity of content of individual frames of the video; determining an acceleration of the content of the individual frames of the video based on at least two of the velocities of the content of the individual frames; determining a score for the individual frames of the video based on the velocity and the acceleration of the content of the individual frames; selecting one or more of the individual frames of the video based on the determined scores; identifying, for individual selected frames, a corresponding video scene, the corresponding video scene comprising a first amount of the video occurring before the selected frame and a second amount of the video occurring after the selected frame; and selecting one or more of the identified video scenes for inclusion in the video summary.