In this paper, we followed the line of predicting ICD codes from unstructured text of the MIMIC dataset (Johnson et al. 2016 ), because it is widely studied and publicly available. The state-of-the-art model for this line of work is the combination of the convolutional neural network (CNN) and the attention mechanism<cite> (Mullenbach et al. 2018)</cite> . However, this model only contains one convolutional layer to build document representations for subsequent layers to predict ICD codes.