1. A method performed by data processing apparatus, the method comprising: receiving a query determined to be a question query and a corresponding answer generated in response to the question query, the answer determined to be responsive to the question query so that it is to be provided to a user device as an answer to the question query, wherein each of the question query and the answer has one or more terms; in response to receiving the question query and the corresponding answer: receiving a first set of rich content items in response to a search process and based on the question query and corresponding answer and ranked according to an order; for each rich content item in the set of first rich content items, receiving data specifying rich content item labels for the rich content item; determining each rich content item having labels that match terms of the answer and terms of the question query as first rich content items; determining each rich content item having labels that match only the terms of the answer or that match only the terms of the question query as second rich content items; demoting the second rich content items in the order in the first set relative to the first rich content items in the first set so that the rich content items in the first set are ranked according to a revised order; selecting, according to the revised order, a rich content item as a selected rich content item to provide with the answer; and providing, to a user device, the answer to the question query and the selected rich content item with the answer to the question query.