1. A method implemented by a data processing apparatus, the method comprising: selecting a mixture of old training data and new training data, the old training data comprising an old text message for which a correct translation to a different language is known, the new training data comprising a new text message for which a correct translation to the different language is not known; sending a plurality of respective requests at different times to a client device of a user, the requests comprising (i) a respective request for the user to translate at least one of the old training data and the new training data and (ii) a respective incentive for the translation; after sending a particular request, receiving a translation from the client device for the old training data of the particular request; comparing the received translation with the correct translation for the old training data; determining an accuracy of the received translation based on the comparison; and updating a confidence score for the user based on the translation, the confidence score representing a likelihood that the user will provide an accurate translation of a text message to the different language at a later time.