1. A method for updating a first speech model in a speech recognition system, comprising: identifying that a user device of a user is in communication with the speech recognition system via a network connection; receiving from the user device, the user device comprising a personal speech model trained for the user through previous speech recognition operations, one or more personal speech model components of the personal speech model trained for the user through previous speech recognition operations, the one or more personal speech model components describing personal speech characteristics of the user, wherein the one or more personal speech model components are received from the user device over the network connection in response to identifying that the user device is in communication with the speech recognition system; updating the first speech model using at least some of the one or more personal speech model components, by modifying at least one speech model component of the first speech model and/or adding at least one speech model component to the first speech model; and performing speech recognition on user speech using the first speech model updated with the at least some of the one or more personal speech model components.