1. A method of segmenting a video sequence, the method comprising: determining a segment score for each of a plurality of multi-scale fixed length segments of the video sequence, each of the segment scores providing a score for a plurality of actions associated with a corresponding multi-scale fixed length segment; selecting a current segment from said plurality of multi-scale fixed length segments of the video sequence; assigning the segment score of a further one of said segments as a context feature, the further segment being disjoint from the current segment and being used to provide context information outside the current segment; determining a variable segment score for the current segment according to the context feature associated with the further segment and the segment score of the current segment; and segmenting the video based on the determined variable segment score for the current segment.