1. A method of performing speech recognition, the method comprising: receiving a speech utterance, spoken by a user, at a service provider from a client device; identifying a time associated with the speech utterance; determining a probability associated with the user and the time, the probability being an expected type of background noise for the user at the time; and based on the probability, applying a background noise model to recognize the speech utterance.