One common approach to measuring compositionality is to compare distributional representations of an MWE and its component words (e.g., Schone and Jurafsky, 2001; Baldwin et al., 2003; Katz and Giesbrecht, 2006; Reddy et al., 2011; Schulte im Walde et al., 2013; <cite>Salehi et al., 2015</cite>) . One issue faced by <cite>such approaches</cite> is that token-level instances of MWEs must be identified in a corpus in order to form distributional representations of them. Although word-level language models are widely used, and their performance can be higher than character-level language models, character-level models have the advantage that they can model out-of-vocabulary words (Mikolov et al., 2012) . In this paper we consider whether character-level neural network language models capture knowledge of MWE compositionality.