1. A method comprising: receiving first audio data corresponding to an utterance of a user; obtaining, from an automated speech recognizer that uses an acoustic model to transcribe utterances, a first transcription of the first audio data; receiving data indicating (i) a selection of one or more terms of the first transcription and (ii) one or more of replacement terms that the user has manually specified as a replacement for the one or more terms; determining a minimum edit distance between the one or more terms of the first transcription and the one or more replacement terms; determining, based at least on the minimum edit distance between the one or more terms of the first transcription and the one or more replacement terms that the user has manually specified as a replacement for the one or more terms, that one or more of the replacement terms that the user has manually specified as a replacement for the one or more terms do not likely represent a correction of one or more of the one or more terms of the first transcription; and in response to determining, based at least on the minimum edit distance between the one or more terms of the first transcription and the one or more replacement terms that the user has manually specified as a replacement for the one or more terms, that the one or more of the replacement terms that the user has manually specified as a replacement for the one or more terms do not likely represent a correction of the one or more terms of the first transcription, bypassing retraining of the acoustic model of the automated speech recognizer using a first portion of the first audio data that is associated with the one or more terms of the first transcription.