1. A method of segmenting a video sequence, the method comprising: determining a segment score for each of a plurality of fixed length segments of the video sequence, each of the segment scores providing a score for a plurality of actions associated with a corresponding fixed length segment; selecting a current segment from said segments of the video sequence; selecting the segment score for a further one of said segments, the further segment being disjoint with the current segment and being used to provide information about actions that were classified outside the current segment; determining a further segment score for the current segment according to the selected segment score; and segmenting the video based on the determined further segment score.