1. A computer-implemented method comprising: receiving a first channel of audio data corresponding to an utterance and a second channel of audio data corresponding to the utterance; generating, using a trained recurrent neural network, (i) a first set of filter parameters for a first filter based on the first channel of audio data and the second channel of audio data and (ii) a second set of filter parameters for a second filter based on the first channel of audio data and the second channel of audio data; generating a single combined channel of audio data by combining (i) audio data of the first channel that has been filtered using the first filter and (ii) audio data of the second channel that has been filtered using the second filter; inputting the audio data for the single combined channel to a neural network trained as an acoustic model; and providing a transcription for the utterance that is determined based at least on output that the neural network provides in response to receiving the combined convolution outputs.