1. A computer implemented method comprising: providing a text-to-speech prompt for output; receiving, at a processing system, particular audio data encoding (i) at least a portion of the text-to-speech prompt and (ii) a user utterance; providing the particular audio data to an additional audio activity detector comprising a model that is trained, using training audio data comprising text-to-speech prompts, to identify whether given audio data comprises additional audio other than a text-to-speech prompt; receiving, from the additional audio activity detector, data indicating that the particular audio data comprises additional audio other than the text-to-speech prompt; and in response to receiving the data indicating that the particular audio data comprises additional audio other than the text-to-speech prompt, initiating a reduction in an audio output level of the text-to-speech prompt.