1. A computer-implemented method for performing speech recognition, comprising: as implemented by one or more computing devices configured to execute specific instructions, receiving audio input; recognizing a first word of an utterance using the audio input, a word network, and a speech unit network, wherein the word network is associated with a language model and the speech unit network is associated with a first speech unit model that comprises pronunciations of words using speech units; and subsequent to recognizing the first word of the utterance: determining information about an accent of a speaker of the audio input using the audio input; obtaining a second speech unit model using the information; associating the second speech unit model with the speech unit network; recognizing a second word of the same utterance using the second speech unit model; and generating speech recognition results comprising the first word, recognized using the first speech unit model, and the second word, recognized using the second speech unit model.