1. A method of using a computer having one or more processors to predict user engagement, comprising: determining, by the one or more processors, engagement metrics indicating user engagement with portions of a plurality of different watched videos; characterizing, by the one or more processors, features from video and/or audio content within the portions of the plurality of different watched videos, the characterized features derived separately from the determined engagement metrics, wherein characterizing the features comprises: identifying a time window of a watched video; and computing a feature vector describing characteristics of content within the watched video within the identified time window; determining, by the one or more processors, a relationship between the characterized features from the content within the portions of the plurality of different watched videos and the engagement metrics of the portions of the plurality of different watched videos; characterizing, by the one or more processors, features from video and/or audio content within a portion of an unwatched video, the unwatched video not among the plurality of different watched videos; predicting, by the one or more processors, an engagement metric for the portion of the unwatched video based at least in part on the characterized features from the content within the portion of the unwatched video and the determined relationship between the characterized features from the content within the portions of the plurality of different watched videos and the engagement metrics of the portions of the plurality of different watched videos; and storing, by the one or more processors, the predicted engagement metric for the portion of the unwatched video.