1. A method of using a computer having one or more processors to predict user engagement, comprising: the one or more processors accomplishing the steps of: determining engagement metrics indicating user engagement with portions of a plurality of different watched videos; characterizing inherent features of video content within the portions of the plurality of different watched videos, the characterizing comprising describing at least one of: frequency coefficients of a visual component of the content, spatial arrangement of color of a visual component of the content, visual texture of a visual component of the content, and motion of pixels in a visual component of the content; determining a relationship between the characterized features of the content within the portions of the plurality of different watched videos and the engagement metrics of the portions of the plurality of different watched videos; characterizing inherent features of video content within a portion of an unwatched video, the unwatched video not among the plurality of watched videos, the characterizing comprising describing at least one of: frequency coefficients of a visual component of the content, spatial arrangement of color of a visual component of the content, visual texture of a visual component of the content, and motion of pixels in a visual component of the content; predicting an engagement metric for the portion of the unwatched video based at least in part on the characterized features of the content within the portion of the unwatched video and the determined relationship between the characterized features of the content within the portions of the plurality of different watched videos and the engagement metrics of the portions of the watched videos; and storing the predicted engagement metric for the portion of the unwatched video.