Similar to <cite>[13]</cite> , we apply a 2DLSTM layer to combine the acoustic model (the LSTM encoder) and the language model (the decoder) without any attention components. Compared to <cite>[13]</cite> , our model is much deeper. We use max-pooling to select the most relevant encoder state whereas <cite>[13]</cite> uses the last horizontal state of the 2DLSTM.