1. A method performed by one or more computers, the method comprising: obtaining n-gram parameter values derived from an n-gram language model, the n-gram parameter values including n-gram parameter values for n-grams that include multiple words; determining n-gram features for a log-linear language model based on the n-grams corresponding to the obtained n-gram parameter values; determining a weight for each of the determined n-gram features, wherein for at least some of the n-gram features, the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram of multiple words, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram; generating a log-linear language model having the determined n-gram features, the determined n-gram features in the log-linear language model having weights that are initialized based on the determined weights; using the log-linear language model to determine a transcription for an utterance; and providing the transcription for the utterance.