We experiment with average pooling and max pooling concatenation over hidden states for semantic prediction, similar to<cite> [Howard and Ruder, 2018]</cite> , and observe increase in model accuracy by 2.2% on sentiment analysis.