1. A computer-implemented method comprising: decoding a first portion of a speech session to generate a first speech recognition hypothesis using a speaker adaptation profile; identifying a segment of the first speech recognition hypothesis; determining that a stability measure for the identified segment satisfies a threshold; triggering a real-time update of the speaker adaptation profile using the identified segment, or using a portion of speech data that corresponds to the identified segment; and decoding a second portion of the speech session to generate a second speech recognition hypothesis using the updated speaker adaptation profile.