Related works have shown that we can increase model capacity, in order to efficiently learn from large speech datasets, by adding more hidden layers rather than making each layer larger. [1]} explored increasing the number of consecutive bidirectional recurrent layers, and [2]} proposed the Deep Speech2, which among a number of optimization techniques, extensively applied batch normalization [3]} to the deep RNNs.
