1. A method comprising: pre-training, by a speech recognition system that includes a language model, the language model on a first set of text training data to obtain first trained parameter values for the language model; pre-training, by the speech recognition system that includes an inverse pronunciation model, the inverse pronunciation model on a second set of text training data that is a different set of text training data than the first set of text training data to obtain first trained parameter values for the inverse pronunciation model, the inverse pronunciation model being a different model than the language model; pre-training, by the speech recognition system that includes an acoustic model, the acoustic model on a first set of audio data to obtain first trained parameter values for the acoustic model, the acoustic model being a different model than the language model and a different model than the inverse pronunciation model; receiving, by the speech recognition system, a second set of audio data that a) comprises a plurality of audio inputs and b) is a different set of audio data than the first set of audio data; training, by the speech recognition system and subsequent to the pre-training, the acoustic model, the inverse pronunciation model, and the language model on the second set of audio data to obtain: second trained parameter values for the acoustic model using the first trained parameter values for the acoustic model, second trained parameter values for the inverse pronunciation model using the first trained parameter values for the inverse pronunciation model, and second trained parameter values for the language model using the first trained parameter values for the language model, the training comprising, for each audio input from the second set of audio data: processing, by the speech recognition system, the respective audio input from the second set of audio data using the first trained parameter values for the acoustic model to generate a respective phoneme score for each of a plurality of phoneme label sequences represented by data in the audio input, wherein the acoustic model comprises one or more long short term memory (LSTM) blocks that receive the audio input and generate an output from the audio input and one or more connectionist temporal classification (CTC) layers that receive the output from the LSTM blocks and transform the output into the phoneme scores; processing, by the speech recognition system, one or more of the phoneme scores using the first trained parameter values for the inverse pronunciation model to generate a respective grapheme score for each of a plurality of grapheme label sequences; processing, by the speech recognition system, one or more of the grapheme scores using the first trained parameter values for the language model to generate a respective text label score for each of a plurality of text label sequences, wherein each text label score reflects a likelihood that the corresponding text label sequence is an accurate transcript of the audio input; and adjusting, by the speech recognition system and using the text label scores for the plurality of text label sequences for the audio input, at least some of the first trained parameter values for the acoustic model, the inverse pronunciation model, and the language model; and causing, by the speech recognition system, storage of the acoustic model that includes the respective second trained parameter values, the inverse pronunciation model that includes the respective second trained parameter values, and the language model that includes the respective second trained parameter values for use together during a speech recognition process.