1. A method performed by data processing apparatus, the method comprising: receiving, by a server system that provides an automated speech recognition service over a computer network, audio data for an utterance detected by a client device in communication with the server system over the computer network; accessing, by the server system, association data that indicates a plurality of associations, each association indicating (i) uncorrupted audio data indicating characteristics of an uncorrupted audio segment, and (ii) a corresponding key based on a corrupted version of the same uncorrupted audio segment, the associations being determined before receiving the audio data for the utterance; selecting, by the server system, uncorrupted audio data based on a comparison of (i) one or more keys based on the audio data for the utterance, with (ii) the keys based on the corrupted audio data; constructing, by the server system, a representation of the utterance comprising the selected uncorrupted audio data; and performing, by the server system, speech recognition on the constructed representation of the utterance to determine a transcription of the utterance.