We believe that the research direction of incorporating language models and multiple levels of representations can help to provide a wide set of rich features that can capture context-dependent semantics as well as linguistic features, such as seen on ELMo <cite>[33]</cite> downstream and linguistic probing task experiments, but for sentence embeddings.