1. A computer-implemented method comprising: receiving, by a computing device that uses voice-based speaker identification, audio data corresponding to an utterance by the user of a predefined hotword; in response to a false rejection of the audio data corresponding to the utterance, prompting the user to verify their identification using a technique other than voice-based speaker identification; in response to the user successfully verifying their identification using the technique other than voice-based speaker identification, prompting the user to confirm that the audio data corresponding to the utterance was falsely rejected; receiving data indicating that the user has confirmed that the audio data corresponding to the utterance was falsely rejected; and in response to receiving the data indicating that the user has confirmed that the audio data corresponding to the utterance was falsely rejected, using the audio data in determining whether audio data corresponding to subsequently received utterances by the user of the predefined hotword are to be accepted or rejected.