1. A method, comprising: providing, by the data processing apparatus to a user device of a user, skippable content items that are each a video that plays before presentation of another video that was selected by the user, that each include an initial portion that is not skippable and a remaining portion that the user can skip after the initial portion of each skippable content item plays on the user device, and that is from a given content category; determining, by the data processing apparatus and for the user, an aggregate engagement level with the skippable content items from the given content category based on the remaining portions of the skippable content items that were presented at the user device based on the user skipping the remaining portions of the skippable content items; determining, by the data processing apparatus and for the user, an interest score for the given category based on a comparison of the aggregate engagement level to a population engagement level with skippable content items from the given content category, the population engagement level reflecting a portion of other users who skip, before other videos selected by each user of the population, the remaining portions of the skippable content items from the given content category; determining, by the data processing apparatus, that the interest score of the user for the given category is below a specified threshold; and in response to receiving a subsequent request for a content item from the given content category and based on determining that the interest score for the given category is below the specified threshold, distributing, to the user device, a non-skippable content item from the given category.