The analyses have focused on wordlevel models, yet character-level models have been shown to outperform word-level models in some NLP tasks, such as text classification (Zhang et al., 2015) , named entity recognition (Kuru et al., 2016) , and time normalization<cite> (Laparra et al., 2018a)</cite> . Thus, there is a need to study pre-trained contextualized character embeddings, to see if they also yield improvements, and if so, to analyze where those benefits are coming from.