Probing Pretrained Language Models for Lexical Semantics

Abstract: The success of large pretrained language models (LMs) such as BERT and RoBERTa has sparked interest in probing their representations, in order to unveil what types of knowledge they implicitly capture. While prior research focused on morphosyntactic, semantic, and world knowledge, it remains unclear to which extent LMs also derive lexical type-level knowledge from words in context. In this work, we present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks, addressing the following questions: 1) How do different lexical knowledge extraction strategies (monolingual versus multilingual source LM, out-of-context versus in-context encoding, inclusion of special tokens, and layer-wise averaging) impact performance? How consistent are the observed effects across tasks and languages? 2) Is lexical knowledge stored in few parameters, or is it scattered throughout the network? 3) How do these representations fare against traditional static word vectors in lexical tasks? 4) Does the lexical information emerging from independently trained monolingual LMs display latent similarities? Our main results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks. Moreover, we validate the claim that lower Transformer layers carry more type-level lexical knowledge, but also show that this knowledge is distributed across multiple layers.

\section{Introduction and Motivation}
\label{s:introduction}
\input{01-introduction}

\section{Lexical Representations from Pretrained Language Models}
\label{s:methodology}
\input{02-methodology}

\section{Experimental Setup}
\label{s:experiments}
\input{03-experiments}

\section{Results and Discussion}
\label{s:results}
\input{04-results}

\section{Further Discussion and Conclusion}
\label{s:conclusion}
\input{05-conclusion}

\section*{Acknowledgments}
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909) awarded to Anna Korhonen. The work of Goran Glavaš and Robert Litschko is supported by the Baden-Württemberg Stiftung (AGREE grant of the Eliteprogramm).

\subsection{Lexical Information in Individual Layers}
\label{ss:individual}
\noindent \textbf{Evaluation Setup.} To better understand which layers contribute the most to the final performance in our lexical tasks, we also probe type-level representations emerging from each individual layer of pretrained LMs. For brevity, we focus on the best performing configurations from previous experiments: \{{\textsc{mono}}\xspace, \textsc{mbert}\}.\{{\textsc{iso}}\xspace, {\textsc{aoc}}\xspace-100\}.{\textsc{nospec}}\xspace.

In addition, tackling Q4 from \S\ref{s:introduction}, we analyse the similarity of representations extracted from  monolingual and multilingual BERT models using the centered kernel alignment (CKA) as proposed by \cite{Kornblith:2019icml}. The linear CKA computes similarity that is invariant to isotropic scaling and orthogonal transformation. It is defined as
{\normalsize
\begin{align}
\text{CKA}(X,Y)=\frac{\norm{Y^\top X}^2_\text{F}}{\left(\norm{X^\top X}_\text{F}\norm{Y^\top Y}_\text{F}\right)}.
\end{align}}%
\noindent $X,Y \in \mathbb{R}^{s \times d}$ are input matrices spanning $s$ $\ell_2$-normalized and mean-centered examples of dimensionality $d=768$. We use CKA in two different experiments: \textbf{1)} measuring \textit{self-similarity} where we compute CKA similarity of representations extracted from different layers for the same word; and \textbf{2)} measuring \textit{bilingual layer correspondence} where we compute CKA similarity of representations extracted from the same layer for two words constituting a translation pair. To this end, we again use BLI dictionaries of \newcite{Glavas:2019acl} (see \S\ref{s:experiments}) covering 7K pairs (training + test pairs).

\vspace{1.8mm}
\noindent \textbf{Discussion.}
Per-layer CKA similarities are provided in Figure~\ref{fig:mono-cka} (self-similarity) and  Figure~\ref{fig:biling-cka} (bilingual), and we show results of representations extracted from individual layers for selected evaluation setups and languages in Table~\ref{tab:per-layer}. We also plot bilingual layer correspondence of true word translations versus randomly paired words for \textsc{en}--\textsc{ru} in Figure~\ref{fig:cka-ru}. Figure~\ref{fig:mono-cka} reveals very similar patterns for both {\textsc{en}}\xspace and {\textsc{fi}}\xspace, and we also observe that self-similarity scores decrease for more distant layers (cf., similarity of $L_1$ and $L_2$ versus $L_1$ and $L_{12}$). However, despite structural similarities identified by linear CKA, the scores from Table~\ref{tab:per-layer} demonstrate that structurally similar layers might encode different amounts of lexical information: e.g., compare performance drops between $L_5$ and $L_8$ in all evaluation tasks.

The results in Table~\ref{tab:per-layer} further suggest that more type-level lexical information is available in lower layers, as all peak scores in the table are achieved with representations extracted from layers $L_1-L_5$. Much lower scores in type-level semantic tasks for higher layers also empirically validate a recent hypothesis of \newcite{Ethayarajh:2019emnlp} ``that contextualised word representations are more context-specific in higher layers.'' We also note that none of the results with L=$n$ configurations from Table~\ref{tab:config} can match best performing \textsc{avg}(L$\leq$\textit{n}) configurations with layer-wise averaging. This confirms our hypothesis that type-level lexical knowledge, although predominantly captured by lower layers, is disseminated across multiple layers, and layer-wise averaging is crucial to uncover that knowledge.

Further, Figure~\ref{fig:biling-cka} and Figure~\ref{fig:cka-ru} reveal that even LMs trained on monolingual data learn similar representations in corresponding layers for word translations (see the {\textsc{mono}}\xspace.{\textsc{aoc}}\xspace columns). Intuitively, this similarity is much more pronounced with {\textsc{aoc}}\xspace configurations with mBERT. The comparison of scores in Figure~\ref{fig:cka-ru} also reveals much higher correspondence scores for true translation pairs than for randomly paired words (i.e., the correspondence scores for random pairings are, as expected, random). Moreover, {\textsc{multi}}\xspace CKA similarity scores turn out to be higher for more similar language pairs (cf. {\textsc{en}}\xspace--{\textsc{de}}\xspace versus {\textsc{en}}\xspace--{\textsc{tr}}\xspace~{\textsc{multi}}\xspace.{\textsc{aoc}}\xspace columns). This suggests that, similar to static WEs, type-level `BERT-based' WEs of different languages also display topological similarity, often termed \textit{approximate isomorphism} \cite{Sogaard:2018acl}, but its degree depends on language proximity. This also clarifies why representations extracted from two independently trained monolingual LMs can be linearly aligned, as validated by BLI and CLIR evaluation (Table~\ref{tab:per-layer} and Figure~\ref{fig:bli-clir-summary}).\footnote{Previous work has empirically validated that sentence representations for semantically similar inputs from different languages are less similar in higher Transformer layers \cite{Singh:2019ws,Wu:2019emnlp}. In Figure~\ref{fig:biling-cka}, we demonstrate that this is also the case for type-level lexical information; however, unlike sentence representations where highest similarity is reported in lowest layers, Figure~\ref{fig:biling-cka} suggests that highest CKA similarities are achieved in intermediate layers $L_5$-$L_8$.}

We also calculated the Spearman's correlation between CKA similarity scores for configurations {\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{100}}\xspace.{\textsc{nospec}}\xspace.\textsc{avg}(L$\leq$\textit{n}), for all $n=0,\ldots,12$, and their  corresponding BLI scores on {\textsc{en}}\xspace--{\textsc{fi}}\xspace, {\textsc{en}}\xspace--{\textsc{de}}\xspace, and {\textsc{de}}\xspace--{\textsc{fi}}\xspace. The correlations are very high: $\rho=1.0, 0.83, 0.99$, respectively. This further confirms the approximate isomorphism hypothesis: it seems that higher structural similarities of representations extracted from monolingual pretrained LMs facilitate their cross-lingual alignment.

\section{Appendix}

URLs to the models and external corpora used in our study are provided in Table~\ref{tab:models} and Table~\ref{tab:excorpora}, respectively. URLs to the evaluation data and task architectures for each evaluation task are provided in Table~\ref{tab:evtasks}. We also report additional and more detailed sets of results across different tasks, word embedding extraction configurations/variants, and language pairs:

\begin{itemize}
    \item In Table~\ref{tab:full-bli1} and Table~\ref{tab:full-bli2}, we provide full BLI results per language pair. All scores are Mean Reciprocal Rank (MRR) scores (in the standard scoring interval, 0.0--1.0).
    \item In Table~\ref{tab:full-clir}, we provide full CLIR results per language pair. All scores are Mean Average Precision (MAP) scores (in the standard scoring interval, 0.0--1.0).
    \item In Table~\ref{tab:full-stm}, we provide full relation prediction (RELP) results for {\textsc{en}}\xspace and {\textsc{de}}\xspace. All scores are micro-averaged $F_1$ scores over 5 runs of the relation predictor \cite{Glavas:2018naacl}. We also report standard deviation for each configuration.
\end{itemize}

Finally, in Figures~\ref{fig:appendix-cka1}-\ref{fig:appendix-cka4}, we also provide heatmaps denoting bilingual layer correspondence, computed via linear CKA similarity \cite{Kornblith:2019icml}, for several \textsc{en}--$L_t$ language pairs (see \S\ref{ss:individual}), which are not provided in the main paper

\onecolumn
\begin{table*}[!t]
\def0.93{0.99}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l X}
\toprule
{\bf Language} & {\bf URL} \\
\cmidrule(lr){2-2}
{{\textsc{en}}\xspace} & {\url{https://huggingface.co/bert-base-uncased}} \\
{{\textsc{de}}\xspace} & {\url{https://huggingface.co/bert-base-german-dbmdz-uncased}} \\
{{\textsc{ru}}\xspace} & {\url{https://huggingface.co/DeepPavlov/rubert-base-cased}} \\
{{\textsc{fi}}\xspace} & {\url{https://huggingface.co/TurkuNLP/bert-base-finnish-uncased-v1}} \\
{{\textsc{zh}}\xspace} & {\url{https://huggingface.co/bert-base-chinese}} \\
{{\textsc{tr}}\xspace} & {\url{https://huggingface.co/dbmdz/bert-base-turkish-uncased}} \\
{Multilingual} & {\url{https://huggingface.co/bert-base-multilingual-uncased}} \\
\cmidrule(lr){2-2}
\multirow{2}{*}{\textsc{it}} & {\url{https://huggingface.co/dbmdz/bert-base-italian-uncased}} \\
& {\url{https://huggingface.co/dbmdz/bert-base-italian-xxl-uncased}} \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{URLs of the models used in our study. The first part of the table refers to the models used in the main experiments throughout the paper, while the second part refers to the models used in side experiments.}
\label{tab:models}
\end{table*}

\begin{table*}[!t]
\def0.93{0.99}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l X}
\toprule
{\bf Language} & {\bf URL} \\
\cmidrule(lr){2-2}
{{\textsc{en}}\xspace} & {\url{http://opus.nlpl.eu/download.php?f=Europarl/v8/moses/de-en.txt.zip}} \\
{{\textsc{de}}\xspace} & {\url{http://opus.nlpl.eu/download.php?f=Europarl/v8/moses/de-en.txt.zip}} \\
{{\textsc{ru}}\xspace} & {\url{http://opus.nlpl.eu/download.php?f=UNPC/v1.0/moses/en-ru.txt.zip}} \\
{{\textsc{fi}}\xspace} & {\url{http://opus.nlpl.eu/download.php?f=Europarl/v8/moses/en-fi.txt.zip}} \\
{{\textsc{zh}}\xspace} & {\url{http://opus.nlpl.eu/download.php?f=UNPC/v1.0/moses/en-zh.txt.zip}} \\
{{\textsc{tr}}\xspace} & {\url{http://data.statmt.org/wmt18/translation-task/news.2017.tr.shuffled.deduped.gz}} \\
 \textsc{it} & {\url{http://opus.nlpl.eu/download.php?f=Europarl/v8/moses/en-it.txt.zip}} \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{Links to the external corpora used in the study. We randomly sample 1M sentences of maximum sequence length 512 from the corresponding corpora.}
\label{tab:excorpora}
\end{table*}
\begin{table*}[!t]
\def0.93{0.99}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l X X}
\toprule
{\bf Task} & {\bf Evaluation Data and/or Model} & {\bf Link} \\
\cmidrule(lr){2-2} \cmidrule(lr){3-3}
{LSIM} & {Multi-SimLex} & {Data: \url{multisimlex.com/}} \\
\cmidrule(lr){2-2} \cmidrule(lr){3-3}
{WA} & {BATS} & {Data: \url{vecto.space/projects/BATS/}} \\
\cmidrule(lr){2-2} \cmidrule(lr){3-3}
\multirow{2}{*}{BLI} & {Data: Dictionaries from \newcite{Glavas:2019acl}} & {Data: \url{github.com/codogogo/xling-eval/tree/master/bli_datasets}} \\
& {Model: VecMap}  & {Model: \url{github.com/artetxem/vecmap}} \\
\cmidrule(lr){2-2} \cmidrule(lr){3-3}
\multirow{2}{*}{CLIR} & {Data: CLEF 2003} & {Data: \url{catalog.elra.info/en-us/repository/browse/ELRA-E0008/}} \\
& {Model: Agg-IDF from \newcite{Litschko:2019sigir}}  & {Model: \url{github.com/rlitschk/UnsupCLIR}} \\
\cmidrule(lr){2-2} \cmidrule(lr){3-3}
\multirow{2}{*}{RELP} & {Data: WordNet-based RELP data} & {Data: \url{github.com/codogogo/stm/tree/master/data/wn-ls}} \\
& {Model: Specialization Tensor Model}  & {Model: \url{github.com/codogogo/stm}} \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{Links to evaluation data and models.}
\label{tab:evtasks}
\end{table*}

\begin{table*}[t]
\def0.93{0.93}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l YYYYYY Y}
\toprule
{\textbf{Configuration}} & {{\textsc{en}}\xspace--{\textsc{de}}\xspace} & {{\textsc{en}}\xspace--{\textsc{tr}}\xspace} & {{\textsc{en}}\xspace--{\textsc{fi}}\xspace} & {{\textsc{en}}\xspace--{\textsc{ru}}\xspace} & {{\textsc{de}}\xspace--{\textsc{tr}}\xspace} & {{\textsc{de}}\xspace--{\textsc{fi}}\xspace} & {{\textsc{de}}\xspace--{\textsc{ru}}\xspace} \\
\cmidrule(lr){2-8}
\textsc{fasttext.wiki} & {0.610} & {0.433} & {0.488} & {0.522} & {0.358} & {0.435} & {0.469} \\
\cmidrule(lr){2-8}
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.390 & 0.332 & 0.392 & 0.409 & 0.237 & 0.269 & 0.291 \\
{\textsc{avg(}L$\leq$4)} & 0.430 & 0.367 & 0.438 & 0.447 & 0.269 & 0.311 & 0.338 \\
{\textsc{avg(}L$\leq$6)} & 0.461 & 0.386 & 0.476 & 0.472 & 0.299 & 0.359 & 0.387 \\
{\textsc{avg(}L$\leq$8)} & 0.472 & 0.390 & 0.486 & 0.487 & 0.315 & 0.387 & 0.407 \\
{\textsc{avg(}L$\leq$10)} & 0.461 & 0.386 & 0.483 & 0.488 & 0.321 & 0.395 & 0.416 \\
{\textsc{avg(}L$\leq$12)} & 0.446 & 0.379 & 0.471 & 0.473 & 0.323 & 0.395 & 0.412 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.399 & 0.342 & 0.386 & 0.403 & 0.242 & 0.269 & 0.292 \\
{\textsc{avg(}L$\leq$4)} & 0.457 & 0.379 & 0.448 & 0.433 & 0.283 & 0.322 & 0.343 \\
{\textsc{avg(}L$\leq$6)} & 0.503 & 0.399 & 0.480 & 0.458 & 0.315 & 0.369 & 0.380 \\
{\textsc{avg(}L$\leq$8)} & 0.527 & 0.414 & 0.499 & 0.461 & 0.332 & 0.394 & 0.391 \\
{\textsc{avg(}L$\leq$10)} & 0.534 & 0.415 & 0.498 & 0.459 & 0.337 & 0.401 & 0.394 \\
{\textsc{avg(}L$\leq$12)} & 0.534 & 0.416 & 0.492 & 0.453 & 0.337 & 0.401 & 0.376 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{100}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.401 & 0.343 & 0.391 & 0.398 & 0.239 & 0.269 & 0.293 \\
{\textsc{avg(}L$\leq$4)} & 0.459 & 0.381 & 0.449 & 0.437 & 0.288 & 0.325 & 0.343 \\
{\textsc{avg(}L$\leq$6)}& 0.504 & 0.403 & 0.484 & 0.459 & 0.318 & 0.373 & 0.382 \\
{\textsc{avg(}L$\leq$8)} & 0.532 & 0.418 & 0.503 & 0.462 & 0.334 & 0.394 & 0.389 \\
{\textsc{avg(}L$\leq$10)} & 0.540 & 0.422 & {0.504} & 0.459 & 0.338 & 0.402 & 0.393 \\
{\textsc{avg(}L$\leq$12)} & 0.542 & 0.426 & 0.500 & 0.454 & 0.343 & 0.401 & 0.378 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.352 & 0.289 & 0.351 & 0.374 & 0.230 & 0.265 & 0.283 \\
{\textsc{avg(}L$\leq$4)} & 0.375 & 0.317 & 0.391 & 0.393 & 0.264 & 0.302 & 0.331 \\
{\textsc{avg(}L$\leq$6)} & 0.386 & 0.330 & 0.406 & 0.407 & 0.289 & 0.350 & 0.376 \\
{\textsc{avg(}L$\leq$8)} & 0.372 & 0.327 & 0.409 & 0.413 & 0.291 & 0.370 & 0.392 \\
{\textsc{avg(}L$\leq$10)} & 0.352 & 0.320 & 0.396 & 0.402 & 0.290 & 0.370 & 0.383 \\
{\textsc{avg(}L$\leq$12)} & 0.313 & 0.310 & 0.373 & 0.394 & 0.283 & 0.358 & 0.371 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{withcls}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.367 & 0.306 & 0.368 & 0.386 & 0.236 & 0.272 & 0.285 \\
{\textsc{avg(}L$\leq$4)} & 0.394 & 0.339 & 0.408 & 0.410 & 0.267 & 0.307 & 0.331 \\
{\textsc{avg(}L$\leq$6)} & 0.406 & 0.344 & 0.428 & 0.425 & 0.294 & 0.353 & 0.381 \\
{\textsc{avg(}L$\leq$8)} & 0.393 & 0.344 & 0.430 & 0.431 & 0.306 & 0.369 & 0.400 \\
{\textsc{avg(}L$\leq$10)} & 0.371 & 0.336 & 0.421 & 0.421 & 0.303 & 0.382 & 0.395 \\
{\textsc{avg(}L$\leq$12)}  & 0.331 & 0.329 & 0.403 & 0.409 & 0.302 & 0.375 & 0.387 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.293 & 0.176 & 0.176 & 0.147 & 0.216 & 0.203 & 0.160 \\
{\textsc{avg(}L$\leq$4)} & 0.304 & 0.184 & 0.190 & 0.164 & 0.219 & 0.214 & 0.178 \\
{\textsc{avg(}L$\leq$6)} & 0.315 & 0.189 & 0.203 & 0.198 & 0.223 & 0.225 & 0.198 \\
{\textsc{avg(}L$\leq$8)} & 0.325 & 0.193 & 0.209 & 0.228 & 0.224 & 0.235 & 0.217 \\
{\textsc{avg(}L$\leq$10)} & 0.330 & 0.194 & 0.210 & 0.243 & 0.220 & 0.234 & 0.226 \\
{\textsc{avg(}L$\leq$12)} & 0.333 & 0.193 & 0.206 & 0.248 & 0.219 & 0.231 & 0.227 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.309 & 0.171 & 0.172 & 0.146 & 0.208 & 0.200 & 0.156 \\
{\textsc{avg(}L$\leq$4)} & 0.350 & 0.186 & 0.189 & 0.186 & 0.224 & 0.214 & 0.191 \\
{\textsc{avg(}L$\leq$6)}  & 0.389 & 0.219 & 0.215 & 0.240 & 0.241 & 0.243 & 0.225 \\
{\textsc{avg(}L$\leq$8)} & 0.432 & 0.246 & 0.251 & 0.287 & 0.255 & 0.263 & 0.254 \\
{\textsc{avg(}L$\leq$10)} & 0.448 & 0.258 & 0.264 & 0.306 & 0.260 & 0.282 & 0.272 \\
{\textsc{avg(}L$\leq$12)} & 0.456 & 0.267 & 0.272 & 0.316 & 0.260 & 0.292 & 0.284 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.292 & 0.173 & 0.175 & 0.143 & 0.209 & 0.203 & 0.154 \\
{\textsc{avg(}L$\leq$4)} & 0.301 & 0.176 & 0.188 & 0.155 & 0.211 & 0.213 & 0.171 \\
{\textsc{avg(}L$\leq$6)} & 0.307 & 0.181 & 0.198 & 0.186 & 0.216 & 0.221 & 0.193 \\
{\textsc{avg(}L$\leq$8)} & 0.315 & 0.184 & 0.202 & 0.207 & 0.213 & 0.228 & 0.208 \\
{\textsc{avg(}L$\leq$10)} & 0.318 & 0.182 & 0.197 & 0.216 & 0.208 & 0.226 & 0.215 \\
{\textsc{avg(}L$\leq$12)} & 0.319 & 0.181 & 0.189 & 0.220 & 0.209 & 0.220 & 0.213 \\
\midrule
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace (\textsc{reverse})} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\geq$12)} & {0.104} & {--} & {0.054} & {--} & {--} & {0.077} & {--} \\
{\textsc{avg(}L$\geq$10)} & {0.119} & {--} & {0.061} & {--} & {--} & {0.063} & {--} \\
{\textsc{avg(}L$\geq$8)} & {0.144} & {--} & {0.108} & {--} & {--} & {0.095} & {--} \\
{\textsc{avg(}L$\geq$6)} & {0.230} & {--} & {0.223} & {--} & {--} & {0.238} & {--} \\
{\textsc{avg(}L$\geq$4)} & {0.308} & {--} & {0.318} & {--} & {--} & {0.335} & {--} \\
{\textsc{avg(}L$\geq$2)} & {0.365} & {--} & {0.385} & {--} & {--} & {0.372} & {--} \\
{\textsc{avg(}L$\geq$0)} & {0.446} & {--} & {0.471} & {--} & {--} & {0.395} & {--} \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{Results in the BLI task across different language pairs and word vector extraction configurations. MRR scores reported. For clarity of presentation, a subset of results is presented in this table, while the rest (and the averages) are presented in Table~\ref{tab:full-bli2}. \textit{\textsc{avg(}L$\leq$n)} means that we average representations over all Transformer layers up to the $n$th layer (included), where $L=0$ refers to the embedding layer, $L=1$ to the bottom layer, and $L=12$ to the final (top) layer. Different configurations are described in \S\ref{s:methodology} and Table~\ref{tab:config}. Additional diagnostic experiments with top-to-bottom layerwise averaging configs (\textsc{reverse}) are run for a subset of languages: \{{\textsc{en}}\xspace, {\textsc{de}}\xspace, {\textsc{fi}}\xspace\}.}
\label{tab:full-bli1}
\end{table*}

\begin{table*}[t]
\def0.93{0.93}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l YYY Y}
\toprule
{\textbf{Configuration}} & {{\textsc{tr}}\xspace--{\textsc{fi}}\xspace} & {{\textsc{tr}}\xspace--{\textsc{ru}}\xspace} & {{\textsc{fi}}\xspace--{\textsc{ru}}\xspace} & {{average}} \\
\cmidrule(lr){2-5}
\textsc{fasttext.wiki} & {0.358} & {0.364} & {0.439} & {0.448} \\
\cmidrule(lr){2-5}
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.237 & 0.217 & 0.290 & 0.306 \\
{\textsc{avg(}L$\leq$4)} & 0.279 & 0.261 & 0.337 & 0.348 \\
{\textsc{avg(}L$\leq$6)} & 0.311 & 0.288 & 0.372 & 0.381 \\
{\textsc{avg(}L$\leq$8)} & 0.334 & 0.315 & 0.387 & 0.398 \\
{\textsc{avg(}L$\leq$10)} & 0.347 & 0.317 & 0.392 & 0.401 \\
{\textsc{avg(}L$\leq$12)} & 0.352 & 0.319 & 0.387 & 0.396 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.247 & 0.221 & 0.284 & 0.308 \\
{\textsc{avg(}L$\leq$4)} & 0.288 & 0.263 & 0.331 & 0.355 \\
{\textsc{avg(}L$\leq$6)} & 0.319 & 0.294 & 0.366 & 0.388 \\
{\textsc{avg(}L$\leq$8)} & 0.334 & 0.311 & 0.375 & 0.404 \\
{\textsc{avg(}L$\leq$10)} & 0.340 & 0.311 & 0.379 & 0.407 \\
{\textsc{avg(}L$\leq$12)} & 0.344 & 0.310 & 0.360 & 0.402 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{100}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.244 & 0.220 & 0.285 & 0.308 \\
{\textsc{avg(}L$\leq$4)} & 0.288 & 0.261 & 0.333 & 0.356 \\
{\textsc{avg(}L$\leq$6)} & 0.322 & 0.291 & 0.367 & 0.390 \\
{\textsc{avg(}L$\leq$8)} & 0.338 & 0.309 & 0.376 & 0.406 \\
{\textsc{avg(}L$\leq$10)} & 0.348 & 0.314 & 0.377 & 0.410 \\
{\textsc{avg(}L$\leq$12)} & 0.349 & 0.311 & 0.361 & 0.407 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.226 & 0.212 & 0.284 & 0.287 \\
{\textsc{avg(}L$\leq$4)} & 0.270 & 0.254 & 0.328 & 0.322 \\
{\textsc{avg(}L$\leq$6)} & 0.302 & 0.274 & 0.358 & 0.348 \\
{\textsc{avg(}L$\leq$8)} & 0.318 & 0.296 & 0.371 & 0.356 \\
{\textsc{avg(}L$\leq$10)} & 0.328 & 0.303 & 0.373 & 0.352 \\
{\textsc{avg(}L$\leq$12)} & 0.328 & 0.306 & 0.368 & 0.340 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{withcls}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.232 & 0.217 & 0.285 & 0.295 \\
{\textsc{avg(}L$\leq$4)} & 0.274 & 0.257 & 0.331 & 0.332 \\
{\textsc{avg(}L$\leq$6)}& 0.307 & 0.279 & 0.362 & 0.358 \\
{\textsc{avg(}L$\leq$8)} & 0.327 & 0.303 & 0.377 & 0.368 \\
{\textsc{avg(}L$\leq$10)} & 0.334 & 0.314 & 0.383 & 0.366 \\
{\textsc{avg(}L$\leq$12)} & 0.340 & 0.317 & 0.373 & 0.357 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.170 & 0.131 & 0.127 & 0.180 \\
{\textsc{avg(}L$\leq$4)} & 0.180 & 0.135 & 0.138 & 0.191 \\
{\textsc{avg(}L$\leq$6)} & 0.188 & 0.147 & 0.151 & 0.204 \\
{\textsc{avg(}L$\leq$8)} & 0.189 & 0.152 & 0.164 & 0.214 \\
{\textsc{avg(}L$\leq$10)} & 0.188 & 0.153 & 0.165 & 0.216 \\
{\textsc{avg(}L$\leq$12)} & 0.188 & 0.158 & 0.163 & 0.217 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)}& 0.165 & 0.127 & 0.130 & 0.178 \\
{\textsc{avg(}L$\leq$4)} & 0.176 & 0.146 & 0.139 & 0.200 \\
{\textsc{avg(}L$\leq$6)} & 0.192 & 0.174 & 0.162 & 0.230 \\
{\textsc{avg(}L$\leq$8)} & 0.210 & 0.192 & 0.185 & 0.258 \\
{\textsc{avg(}L$\leq$10)} & 0.219 & 0.198 & 0.200 & 0.271 \\
{\textsc{avg(}L$\leq$12)} & 0.223 & 0.198 & 0.206 & 0.277 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.163 & 0.126 & 0.123 & 0.176 \\
{\textsc{avg(}L$\leq$4)} & 0.175 & 0.128 & 0.133 & 0.185 \\
{\textsc{avg(}L$\leq$6)} & 0.179 & 0.139 & 0.142 & 0.196 \\
{\textsc{avg(}L$\leq$8)} & 0.182 & 0.144 & 0.152 & 0.203 \\
{\textsc{avg(}L$\leq$10)} & 0.178 & 0.141 & 0.153 & 0.203 \\
{\textsc{avg(}L$\leq$12)} & 0.175 & 0.143 & 0.150 & 0.202 \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{Results in the bilingual lexicon induction (BLI) task across different language pairs and word vector extraction configurations: Part II. MAP scores reported. For clarity of presentation, a subset of results is presented in this table, while the rest (also used to calculate the averages) is provided in Table~\ref{tab:full-bli1} in the previous page. \textit{\textsc{avg(}L$\leq$n)} means that we average representations over all Transformer layers up to the $n$th layer (included), where $L=0$ refers to the embedding layer, $L=1$ to the bottom layer, and $L=12$ to the final (top) layer. Different configurations are described in \S\ref{s:methodology} and Table~\ref{tab:config}.}
\label{tab:full-bli2}
\end{table*}

\begin{table*}[t]
\def0.93{0.93}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l YYYYYY Y}
\toprule
{\textbf{Configuration}} & {{\textsc{en}}\xspace--{\textsc{de}}\xspace} & {{\textsc{en}}\xspace--{\textsc{fi}}\xspace} & {{\textsc{en}}\xspace--{\textsc{ru}}\xspace} & {{\textsc{de}}\xspace--{\textsc{fi}}\xspace} & {{\textsc{de}}\xspace--{\textsc{ru}}\xspace} & {{\textsc{fi}}\xspace--{\textsc{ru}}\xspace} & {{average}} \\
\cmidrule(lr){2-8}
\textsc{fasttext.wiki} & {\bf 0.193} & {\bf 0.136} & {0.118} & {\bf 0.221} & {\bf 0.112} & {0.105} & {\bf 0.148} \\
\cmidrule(lr){2-8}
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.059 & 0.075 & 0.106 & 0.126 & 0.086 & 0.123 & 0.096 \\
{\textsc{avg(}L$\leq$4)} & 0.061 & 0.069 & 0.098 & 0.111 & 0.075 & 0.106 & 0.087 \\
{\textsc{avg(}L$\leq$6)} & 0.052 & 0.061 & 0.079 & 0.112 & 0.068 & 0.102 & 0.079 \\
{\textsc{avg(}L$\leq$8)}& 0.042 & 0.048 & 0.075 & 0.112 & 0.063 & 0.105 & 0.074 \\
{\textsc{avg(}L$\leq$10)} & 0.036 & 0.043 & 0.067 & 0.107 & 0.065 & 0.080 & 0.066 \\
{\textsc{avg(}L$\leq$12)} & 0.032 & 0.034 & 0.059 & 0.097 & 0.077 & 0.083 & 0.064 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.069 & 0.078 & 0.094 & 0.109 & 0.078 & 0.108 & 0.089 \\
{\textsc{avg(}L$\leq$4)} & 0.076 & 0.105 & 0.119 & 0.112 & 0.098 & 0.117 & 0.104 \\
{\textsc{avg(}L$\leq$6)} & 0.086 & 0.090 & 0.129 & 0.122 & 0.098 & 0.125 & 0.108 \\
{\textsc{avg(}L$\leq$8)} & 0.092 & 0.073 & 0.137 & 0.105 & 0.100 & 0.114 & 0.103 \\
{\textsc{avg(}L$\leq$10)} & 0.095 & 0.073 & 0.147 & 0.102 & 0.102 & 0.135 & 0.109 \\
{\textsc{avg(}L$\leq$12)} & 0.104 & 0.073 & 0.139 & 0.100 & 0.105 & 0.131 & 0.109 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{100}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.073 & 0.081 & 0.097 & 0.111 & 0.078 & 0.106 & 0.091 \\
{\textsc{avg(}L$\leq$4)}& 0.078 & 0.107 & 0.115 & 0.107 & 0.100 & 0.115 & 0.104 \\
{\textsc{avg(}L$\leq$6)} & 0.087 & 0.087 & 0.127 & 0.132 & 0.103 & {\bf 0.123} & 0.110 \\
{\textsc{avg(}L$\leq$8)} & 0.091 & 0.076 & 0.137 & 0.118 & 0.101 & 0.106 & 0.105 \\
{\textsc{avg(}L$\leq$10)} & 0.099 & 0.074 & {\bf 0.161} & 0.103 & 0.104 & 0.104 & 0.107 \\
{\textsc{avg(}L$\leq$12)} & 0.106 & 0.076 & 0.146 & 0.105 & 0.106 & 0.100 & 0.106 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.044 & 0.045 & 0.076 & 0.095 & 0.067 & 0.098 & 0.071 \\
{\textsc{avg(}L$\leq$4)} & 0.039 & 0.042 & 0.079 & 0.094 & 0.066 & 0.100 & 0.070 \\
{\textsc{avg(}L$\leq$6)} & 0.024 & 0.034 & 0.069 & 0.089 & 0.066 & 0.094 & 0.063 \\
{\textsc{avg(}L$\leq$8)} & 0.018 & 0.020 & 0.039 & 0.068 & 0.059 & 0.092 & 0.049 \\
{\textsc{avg(}L$\leq$10)} & 0.016 & 0.016 & 0.030 & 0.048 & 0.058 & 0.067 & 0.039 \\
{\textsc{avg(}L$\leq$12)}& 0.014 & 0.013 & 0.033 & 0.034 & 0.064 & 0.061 & 0.036 \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{withcls}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.050 & 0.057 & 0.086 & 0.106 & 0.071 & 0.108 & 0.080 \\
{\textsc{avg(}L$\leq$4)} & 0.046 & 0.055 & 0.084 & 0.104 & 0.071 & 0.102 & 0.077 \\
{\textsc{avg(}L$\leq$6)} & 0.032 & 0.042 & 0.076 & 0.103 & 0.066 & 0.097 & 0.069 \\
{\textsc{avg(}L$\leq$8)} & 0.025 & 0.028 & 0.046 & 0.086 & 0.059 & 0.101 & 0.057 \\
{\textsc{avg(}L$\leq$10)} & 0.021 & 0.030 & 0.037 & 0.072 & 0.057 & 0.079 & 0.049 \\
{\textsc{avg(}L$\leq$12)} & 0.020 & 0.016 & 0.032 & 0.052 & 0.045 & 0.072 & 0.040 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.110 & 0.009 & 0.045 & 0.057 & 0.020 & 0.013 & 0.042 \\
{\textsc{avg(}L$\leq$4)} & 0.100 & 0.007 & 0.075 & 0.044 & 0.025 & 0.011 & 0.044 \\
{\textsc{avg(}L$\leq$6)}& 0.098 & 0.007 & 0.046 & 0.043 & 0.029 & 0.030 & 0.042 \\
{\textsc{avg(}L$\leq$8)} & 0.088 & 0.008 & 0.052 & 0.043 & 0.032 & 0.031 & 0.042 \\
{\textsc{avg(}L$\leq$10)} & 0.084 & 0.008 & 0.051 & 0.042 & 0.034 & 0.026 & 0.041 \\
{\textsc{avg(}L$\leq$12)} & 0.082 & 0.006 & 0.048 & 0.039 & 0.037 & 0.024 & 0.039 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.127 & 0.013 & 0.049 & 0.027 & 0.019 & 0.009 & 0.041 \\
{\textsc{avg(}L$\leq$4)} & 0.123 & 0.018 & 0.055 & 0.032 & 0.029 & 0.008 & 0.044 \\
{\textsc{avg(}L$\leq$6)} & 0.120 & 0.018 & 0.055 & 0.051 & 0.042 & 0.009 & 0.049 \\
{\textsc{avg(}L$\leq$8)} & 0.123 & 0.018 & 0.057 & 0.053 & 0.049 & 0.016 & 0.053 \\
{\textsc{avg(}L$\leq$10)} & 0.127 & 0.019 & 0.062 & 0.050 & 0.051 & 0.018 & 0.054 \\
{\textsc{avg(}L$\leq$12)} & 0.128 & 0.021 & 0.065 & 0.049 & 0.052 & 0.019 & 0.056 \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & 0.072 & 0.005 & 0.032 & 0.014 & 0.016 & 0.004 & 0.024 \\
{\textsc{avg(}L$\leq$4)} & 0.075 & 0.004 & 0.027 & 0.014 & 0.022 & 0.005 & 0.024 \\
{\textsc{avg(}L$\leq$6)} & 0.065 & 0.004 & 0.026 & 0.015 & 0.027 & 0.007 & 0.024 \\
{\textsc{avg(}L$\leq$8)} & 0.054 & 0.004 & 0.035 & 0.015 & 0.032 & 0.008 & 0.025 \\
{\textsc{avg(}L$\leq$10)} & 0.054 & 0.005 & 0.032 & 0.017 & 0.035 & 0.007 & 0.025 \\
{\textsc{avg(}L$\leq$12)} & 0.058 & 0.004 & 0.034 & 0.018 & 0.032 & 0.006 & 0.025 \\
\midrule
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace (\textsc{reverse})} & {} & {} & {} & {} & {} & {} & {} \\
{\textsc{avg(}L$\geq$12)} & {0.005} & {0.012} & {--} & {0.001} & {--} & {--} & {--} \\
{\textsc{avg(}L$\geq$10)} & {0.002} & {0.002} & {--} & {0.001} & {--} & {--} & {--} \\
{\textsc{avg(}L$\geq$8)} & {0.004} & {0.002} & {--} & {0.002} & {--} & {--} & {--} \\
{\textsc{avg(}L$\geq$6)} & {0.014} & {0.006} & {--} & {0.004} & {--} & {--} & {--} \\
{\textsc{avg(}L$\geq$4)} & {0.020} & {0.012} & {--} & {0.016} & {--} & {--} & {--} \\
{\textsc{avg(}L$\geq$2)} & {0.024} & {0.019} & {--} & {0.043} & {--} & {--} & {--} \\
{\textsc{avg(}L$\geq$0)} & {0.032} & {0.034} & {--} & {0.097} & {--} & {--} & {--} \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{Results in the CLIR task across different language pairs and word vector extraction configurations. MAP scores reported; \textit{\textsc{avg(}L$\leq$n)} means that we average representations over all Transformer layers up to the $n$th layer (included), where $L=0$ refers to the embedding layer, $L=1$ to the bottom layer, and $L=12$ to the final (top) layer. Different configurations are described in \S\ref{s:methodology} and Table~\ref{tab:config}. Additional diagnostic experiments with top-to-bottom layerwise averaging configs (\textsc{reverse}) are run for a subset of languages: \{{\textsc{en}}\xspace, {\textsc{de}}\xspace, {\textsc{fi}}\xspace\}.}
\label{tab:full-clir}
\end{table*}

\begin{table*}[t]
\def0.93{0.93}
\centering
{\footnotesize
\begin{adjustbox}{max width=\linewidth}
\begin{tabularx}{\textwidth}{l YY}
\toprule
{\textbf{Configuration}} & {{\textsc{en}}\xspace} & {{\textsc{de}}\xspace} \\
\cmidrule(lr){2-3}
\textsc{fasttext.wiki} & {$0.660_{\pm 0.008}$} & {$0.601_{\pm 0.007}$} \\
\textsc{random.xavier} & {$0.473_{\pm 0.003}$} & {$0.512_{\pm 0.008}$} \\
\cmidrule(lr){2-3}
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.688_{\pm 0.007}$} & {$0.649_{\pm 0.002}$} \\
{\textsc{avg(}L$\leq$4)} & {$0.698_{\pm 0.002}$} & {$0.664_{\pm 0.004}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.699_{\pm 0.007}$} & {$0.677_{\pm 0.006}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.706_{\pm 0.003}$} & {$0.674_{\pm 0.016}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.718_{\pm 0.002}$} & {$0.679_{\pm 0.008}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.714_{\pm 0.012}$} & {$0.673_{\pm 0.003}$} \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)}  & {$0.690_{\pm 0.007}$} & {$0.657_{\pm 0.005}$} \\
{\textsc{avg(}L$\leq$4)} & {$0.705_{\pm 0.006}$} & {$0.671_{\pm 0.009}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.714_{\pm 0.008}$} & {$0.675_{\pm 0.014}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.722_{\pm 0.004}$} & {$0.681_{\pm 0.010}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.719_{\pm 0.007}$} & {$0.682_{\pm 0.007}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.720_{\pm 0.005}$} & {$0.680_{\pm 0.007}$} \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{aoc}}\xspace-{\textsc{100}}\xspace.{\textsc{nospec}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.692_{\pm 0.007}$} & {$0.655_{\pm 0.007}$} \\
{\textsc{avg(}L$\leq$4)} & {$0.709_{\pm 0.007}$} & {$0.670_{\pm 0.005}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.718_{\pm 0.009}$} & {$0.672_{\pm 0.008}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.717_{\pm 0.003}$} & {$0.680_{\pm 0.006}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.721_{\pm 0.009}$} & {$0.678_{\pm 0.004}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.715_{\pm 0.003}$} & {$0.678_{\pm 0.006}$} \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.688_{\pm 0.008}$} & {$0.654_{\pm 0.012}$} \\
{\textsc{avg(}L$\leq$4)} & {$0.698_{\pm 0.011}$} & {$0.662_{\pm 0.008}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.711_{\pm 0.005}$} & {$0.664_{\pm 0.005}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.709_{\pm 0.008}$} & {$0.663_{\pm 0.015}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.712_{\pm 0.006}$} & {$0.669_{\pm 0.003}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.704_{\pm 0.005}$} & {$0.666_{\pm 0.013}$} \\
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{withcls}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.693_{\pm 0.004}$} & {$0.649_{\pm 0.016}$} \\
{\textsc{avg(}L$\leq$4)} & {$0.699_{\pm 0.004}$} & {$0.664_{\pm 0.006}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.709_{\pm 0.002}$} & {$0.671_{\pm 0.006}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.710_{\pm 0.003}$} & {$0.679_{\pm 0.006}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.713_{\pm 0.006}$} & {$0.670_{\pm 0.007}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.705_{\pm 0.005}$} & {$0.676_{\pm 0.006}$} \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.671_{\pm 0.009}$} & {$0.628_{\pm 0.013}$} \\
{\textsc{avg(}L$\leq$4)} & {$0.669_{\pm 0.006}$} & {$0.640_{\pm 0.004}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.684_{\pm 0.010}$} & {$0.637_{\pm 0.009}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.680_{\pm 0.005}$} & {$0.647_{\pm 0.006}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.676_{\pm 0.006}$} & {$0.629_{\pm 0.008}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.681_{\pm 0.005}$} & {$0.637_{\pm 0.004}$} \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{aoc}}\xspace-{\textsc{10}}\xspace.{\textsc{nospec}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.674_{\pm 0.005}$} & {$0.635_{\pm 0.011}$} \\
{\textsc{avg(}L$\leq$4)}& {$0.681_{\pm 0.006}$} & {$0.630_{\pm 0.007}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.692_{\pm 0.008}$} & {$0.649_{\pm 0.010}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.695_{\pm 0.004}$} & {$0.652_{\pm 0.011}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.704_{\pm 0.005}$} & {$0.657_{\pm 0.012}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.702_{\pm 0.005}$} & {$0.661_{\pm 0.008}$} \\
\rowcolor{Gray}
{{\textsc{multi}}\xspace.{\textsc{iso}}\xspace.{\textsc{all}}\xspace} & {} & {} \\
{\textsc{avg(}L$\leq$2)} & {$0.674_{\pm 0.004}$} & {$0.626_{\pm 0.014}$} \\
{\textsc{avg(}L$\leq$4)}& {$0.682_{\pm 0.009}$} & {$0.640_{\pm 0.009}$} \\
{\textsc{avg(}L$\leq$6)} & {$0.680_{\pm 0.002}$} & {$0.632_{\pm 0.007}$} \\
{\textsc{avg(}L$\leq$8)} & {$0.683_{\pm 0.003}$} & {$0.638_{\pm 0.010}$} \\
{\textsc{avg(}L$\leq$10)} & {$0.678_{\pm 0.007}$} & {$0.638_{\pm 0.015}$} \\
{\textsc{avg(}L$\leq$12)} & {$0.676_{\pm 0.013}$} & {$0.636_{\pm 0.005}$} \\
\midrule
\rowcolor{Gray}
{{\textsc{mono}}\xspace.{\textsc{iso}}\xspace.{\textsc{nospec}}\xspace (\textsc{reverse})} & {} & {} \\
{\textsc{avg(}L$\geq$12)} & {$0.683_{\pm 0.007}$} & {$0.628_{\pm 0.009}$} \\
{\textsc{avg(}L$\geq$10)} & {$0.692_{\pm 0.014}$} & {$0.628_{\pm 0.008}$} \\
{\textsc{avg(}L$\geq$8)} & {$0.688_{\pm 0.016}$} & {$0.648_{\pm 0.007}$} \\
{\textsc{avg(}L$\geq$6)} & {$0.704_{\pm 0.015}$} & {$0.658_{\pm 0.006}$} \\
{\textsc{avg(}L$\geq$4)} & {$0.704_{\pm 0.008}$} & {$0.668_{\pm 0.007}$} \\
{\textsc{avg(}L$\geq$2)} & {$0.707_{\pm 0.008}$} & {$0.667_{\pm 0.004}$} \\
{\textsc{avg(}L$\geq$0)} & {$0.714_{\pm 0.012}$} & {$0.673_{\pm 0.003}$} \\
\bottomrule
\end{tabularx}
\end{adjustbox}
}
\vspace{-1.5mm}
\caption{Results in the relation prediction task (RELP) across different word vector extraction configurations. Micro-averaged $F_1$ scores reported    , obtained as averages over 5 experimental runs for each configuration; standard deviation is also reported. \textit{\textsc{avg(}L$\leq$n)} means that we average representations over all Transformer layers up to the $n$th layer (included), where $L=0$ refers to the embedding layer, $L=1$ to the bottom layer, and $L=12$ to the final (top) layer. Different configurations are described in \S\ref{s:methodology} and Table~\ref{tab:config}. \textsc{random.xavier} are 768-dim vectors for the same vocabularies, randomly initialised via Xavier initialisation \cite{Glorot:2010aistats}.}
\label{tab:full-stm}
\end{table*}

\clearpage

\begin{figure*}[!t]
    \centering
    \begin{subfigure}[!ht]{0.488\linewidth}
        \centering
        \includegraphics[width=0.98\linewidth]{./en-de}

        \caption{{\textsc{en}}\xspace--{\textsc{de}}\xspace: Word translation pairs}
        \label{fig:en-de}
    \end{subfigure}
    \begin{subfigure}[!ht]{0.488\textwidth}
        \centering
        \includegraphics[width=0.98\linewidth]{./en-de-random}

        \caption{{\textsc{en}}\xspace--{\textsc{de}}\xspace: Random word pairs}
        \label{fig:en-de-random}
    \end{subfigure}
    \vspace{-2mm}
    \caption{CKA similarity scores of type-level word representations extracted from each layer (using different extraction configurations, see Table~\ref{tab:config}) for a set of \textbf{(a)} 7K {\textsc{en}}\xspace--{\textsc{de}}\xspace translation pairs from the BLI dictionaries of \newcite{Glavas:2019acl}; \textbf{(b)} 7K random {\textsc{en}}\xspace--{\textsc{de}}\xspace pairs.}
    \vspace{-1.5mm}
\label{fig:appendix-cka1}
\end{figure*}

\begin{figure*}[!t]
    \centering
    \begin{subfigure}[!ht]{0.488\linewidth}
        \centering
        \includegraphics[width=0.98\linewidth]{./en-fi}

        \caption{{\textsc{en}}\xspace--{\textsc{fi}}\xspace: Word translation pairs}
        \label{fig:en-fi}
    \end{subfigure}
    \begin{subfigure}[!ht]{0.488\textwidth}
        \centering
        \includegraphics[width=0.98\linewidth]{./en-fi-random}

        \caption{{\textsc{en}}\xspace--{\textsc{fi}}\xspace: Random word pairs}
        \label{fig:en-fi-random}
    \end{subfigure}
    \vspace{-2mm}
    \caption{CKA similarity scores of type-level word representations extracted from each layer (using different extraction configurations, see Table~\ref{tab:config}) for a set of \textbf{(a)} 7K {\textsc{en}}\xspace--{\textsc{fi}}\xspace translation pairs from the BLI dictionaries of \newcite{Glavas:2019acl}; \textbf{(b)} 7K random {\textsc{en}}\xspace--{\textsc{fi}}\xspace pairs.}
    \vspace{-1.5mm}
\label{fig:appendix-cka3}
\end{figure*}

\begin{figure*}[!t]
    \centering
    \begin{subfigure}[!ht]{0.488\linewidth}
        \centering
        \includegraphics[width=0.98\linewidth]{./en-tr}

        \caption{{\textsc{en}}\xspace--{\textsc{tr}}\xspace: Word translation pairs}
        \label{fig:en-tr}
    \end{subfigure}
    \begin{subfigure}[!ht]{0.488\textwidth}
        \centering
        \includegraphics[width=0.98\linewidth]{./en-tr-random}

        \caption{{\textsc{en}}\xspace--{\textsc{tr}}\xspace: Random word pairs}
        \label{fig:en-tr-random}
    \end{subfigure}
    \vspace{-2mm}
    \caption{CKA similarity scores of type-level word representations extracted from each layer (using different extraction configurations, see Table~\ref{tab:config}) for a set of \textbf{(a)} 7K {\textsc{en}}\xspace--{\textsc{tr}}\xspace translation pairs from the BLI dictionaries of \newcite{Glavas:2019acl}; \textbf{(b)} 7K random {\textsc{en}}\xspace--{\textsc{tr}}\xspace pairs.}
    \vspace{-1.5mm}
\label{fig:appendix-cka4}
\end{figure*}


A sensitive search for CO emission from faint blue galaxies at z~0.5

Abstract: We have obtained sensitive upper limits on the CO J=2-1 and CO J=3-2 emission lines for five faint blue galaxies with redshifts $z\sim 0.5$ using the IRAM 30~m telescope. These observations would have been able to detect the luminous infrared galaxy IRAS F10214+4724 if it were located at this redshift and unlensed. However, they are not sensitive enough to detect the prototype starburst galaxy M82 or the HII galaxy UM448 if they were located at this redshift. Our upper limits for the CO emission are consistent with between 19% and 66% of the total galactic mass being in the form of molecular hydrogen, and thus shed little light on the ultimate fate of these galaxies.

\section{Introduction}

A large population of faint blue galaxies
at moderate redshifts was first identified by Tyson (\cite{tyson}).
These galaxies are puzzling both because of their large
space density (30 times that of ordinary bright galaxies,
Lilly et al. \cite{lilly}; Babul \& Rees \cite{babul}) and
their lack of obvious bright counterparts in the local universe.
Recent studies have provided us with a detailed look at a sample
of these galaxies at $z=0.1-0.6$ (Koo et al. \cite{koo94}, \cite{koo95};
Guzman et al. \cite{guzman}). The galaxies have star formation rates of
1-20 M$_\odot$ yr$^{-1}$ (Koo et al. \cite{koo95}),
comparable to the total star formation rates of present
day disk galaxies (Kennicutt \cite{kennicutt}).
However, they are very
compact, with half-maximum diameters of only 2-4 kpc (Koo et al. \cite{koo94}).
Although their blue luminosities are also comparable to local field galaxies
($M_B \sim -21$), their total masses are estimated at only $1-5\times 10^9$
M$_\odot$, much smaller than the typical masses of normal elliptical
or spiral galaxies ($\sim 10^{11}$ M$_\odot$) (Guzman et al. \cite{guzman}).
These observations suggest the galaxies are  dwarf galaxies
near their peak luminosity after undergoing a major burst of star formation.
Since star formation is intricately linked with the presence of molecular gas
in the local universe, these galaxies are likely to contain
significant amounts of molecular gas to provide the fuel for the
observed starbursts.

The detection of CO at $z=2.28$ in the luminous infrared galaxy
IRAS F10214+4724 (Brown \& Vanden Bout \cite{brown}; Solomon et al.
\cite{sol92}) has
stimulated numerous searches for CO emission from galaxies at moderate
to high redshifts (Wiklind \& Combes \cite{wik94b}; Evans et al. \cite{evans}).
However, these searches were less successful than expected, and only highly
amplified objects have been detected: the Cloverleaf quasar at $z=2.56$
(Barvainis et al. \cite{barvainis}) and BR1202-0725 at $z=4.69$ (Ohta et al.
\cite{ohta}; Omont et
al. \cite{omont}).
Recently,
Scoville et al. (\cite{scoville}) reported the detection of the first
non-lensed object at $z=2.394$, the weak radio galaxy 53W002.
The derived molecular mass is so high
($7.4 \times 10^{10}$ M$_\odot$ with a standard CO-to-H$_{2}$
conversion factor, and even more
if the metallicity is low) that it
constitutes between 30 and 80\% of the total dynamical mass,
depending on the unknown inclination.
The lensing that can amplify CO emission to a level that is detectable
with current instruments can also be produced
by galaxy clusters. A search for CO emission from four giant arcs
in clusters has been reported by Casoli et al. (\cite{casoli}),
and they detected
one of them at $z= 0.725$. Another sensitive technique
for probing the cold molecular interstellar medium
at high redshifts is via absorption lines
(e.g. Combes \& Wiklind \cite{combes}).
Four systems have been detected in more than a dozen molecules in absorption
with $z$ between 0.25 and 0.9
(Wiklind \& Combes \cite{wik94a}, \cite{wik95}, \cite{wik96a}, \cite{wik96b}).

In this letter, we present the results of
a survey for CO emission from five
faint blue galaxies at redshifts $z\sim 0.5$ selected from
the sample of Koo et al. (\cite{koo95}). This survey differs from
previous surveys in targeting galaxies with known
star formation rates, as opposed to damped Lyman $\alpha$
systems (Wiklind \& Combes \cite{wik94b}) or distant radio galaxies
(Evans et al. \cite{evans}). Unfortunately, as in previous surveys,
we have achieved only upper limits to the CO
emission from our target objects.

\section{Observations and data reduction}

\subsection{Observations}

   \begin{figure}
\psfig{figure=koo_f1.ps,width=8.5cm,bbllx=39mm,bblly=43mm,bburx=137mm,bbury=244mm}
     \caption[]{$^{12}$CO J=2-1 and J=3-2 spectra for the five
faint blue galaxies. The observed CO transition is marked in each
spectrum. The temperature scale is T$_{mb}$ in mK and
the velocity resolution is 10 km s$^{-1}$. No galaxies
were detected.}\label{fig-1}
    \end{figure}

Of the 17 galaxies in the Koo et al. (\cite{koo95})
sample of faint blue galaxies,
nine have redshifts such that the CO J=2-1 and J=3-2 lines
are accessible with the IRAM receivers. Three of these nine galaxies
are somewhat more extended than the other six and indeed
appear non-stellar in ground-based optical observations (Koo et al.
\cite{koo95}). We excluded these three galaxies from our sample
on the grounds that they may be somewhat more massive objects,
akin perhaps to small spiral galaxies.
Observations of five of the remaining six galaxies
were obtained with the IRAM 30 m telescope in two separate
observing runs in January 1996 and June 1997.  The sixth galaxy,
SA57-5482, was not observed due to time constraints.
The half-power beam width is 17$^{\prime\prime}$
at 2--mm and 12$^{\prime\prime}$ at 1.3--mm. We used the 2-- and 1.3--mm  SiS
receivers to observe both lines simultaneously. The receivers were
all used in single sideband mode, and the typical system temperatures
were 250-500 K for the 2--mm receiver and 350-700 K for the 1.3--mm receiver,
in T$_A^*$ scale (or, on average, 600 K
and 1200 K in T$_{MB}$ scale, respectively).
The backends were essentially two 1MHz-filter-banks, of 512 channels each,
and in addition an auto-correlator; the spectra have been smoothed to
10 km s$^{-1}$ resolution. The observations
were made using a nutating secondary with a beam throw of 1.5$^\prime$.
The pointing was checked every two hours and the pointing accuracy was
estimated to be 3$^{\prime\prime}$ rms.

\subsection{Data reduction and analysis}

   \begin{table*}
      \caption[]{CO upper limits for faint blue galaxies}
\label{tbl-1}
\begin{flushleft}
\begin{tabular}{llllllllll}
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Galaxy & RA(1950) & DEC(1950) & z & $\Delta V$ &
time & $\sigma(2-1)$ & $\sigma(3-2)$
& $I_{CO}(3-2)$ & $L_{CO}(3-2)$ \\
& ($h$ $m$ $s$) & ($^o$ $^\prime$ $^{\prime\prime}$) & & (km s$^{-1}$)
& (hr) & (mK) & (mK) &
(K km s$^{-1}$) & (K km s$^{-1}$ pc$^2$) \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
SA57-1501 &13:05:43.9 &29:24:59 &0.4993 & 143 & 9.3 & 2.2 & 3.7 & 0.42 & 2.6$\times 10^9$ \\
SA57-7042 &13:05:03.2 &29:34:26 &0.5250 & 273 & 2.4 & 6.3 & 10.8 & 1.69 & 1.1$\times 10^{10}$\\
SA57-10601 &13:06:25.0 &29:39:39 &0.4384 & 101 & 1.8 & 5.3 & 5.5 & 0.52 & 2.8$\times 10^9$\\
SA57-17731 &13:06:15.5 &29:50:47 &0.6612 & 242 & 3.9 & 3.7 & 5.9 & 0.87 & 7.4$\times 10^9$ \\
Herc-1-13088 &17:19:04.4 &50:04:00 &0.4357 & 106 & 4.3 & 6.8 & 9.7 & 0.95 & 5.1$\times 10^9$
\\
\noalign{\smallskip}
\hline
\end{tabular}
\end{flushleft}
\begin{list}{}{}
\item[] Temperatures are given in the main beam scale
\end{list}
   \end{table*}

The spectra were first inspected, and any spectrum showing baseline
curvature or other artifacts was discarded.
The remaining spectra were averaged together, weighted by their rms noise.
A first order baseline was removed from each average spectrum, and
the spectra were smoothed
to a resolution of 10 km s$^{-1}$ to produce
the final spectra (Fig.~\ref{fig-1}).
The final temperatures (and rms noise in Table 1)
have been converted to the $T_{MB}$ temperature scale ($\eta_{MB} = 0.45$
at 230 GHz, 0.59 at 150 GHz).

Upper limits to the integrated CO intensity were derived using the
rms noise measured from the CO spectra and the velocity widths
obtained from measurements of optical emission lines (Koo et al. \cite{koo95}).
We adopt as the 3$\sigma$ upper limit to the CO intensity
$$I_{CO} \le {{3\sigma \Delta V} \over {\sqrt{N_{chan}}}}
\hskip6pt \rm{K\hskip3pt km \hskip3pt s^{-1}}$$
(Wiklind \& Combes \cite{wik94b}),
where $\sigma$ is the rms noise in K measured in our 10 km s$^{-1}$
channels, $\Delta V$ is the velocity width of the CO line, here
taken to be the full-width half-maximum of the optical lines,
and $N_{chan} = \Delta V/10$ km s$^{-1}$ is the number of channels in the
velocity width.
The CO luminosity for a source at high redshift is given by
$$L_{CO} = 23.5 I_{CO} \Omega_B {{D_L^2}\over {(1+z)^3}} \hskip6pt
\rm{K\hskip3pt km \hskip3pt s^{-1}\hskip3pt pc^2}$$
where $\Omega_B$ is the area of the main beam in square arcseconds
and $D_L = (c/H_o q_o^2)[q_o z + (q_o-1)(\sqrt{1+2q_oz}-1)]$
is the luminosity distance in Mpc (Wiklind \& Combes \cite{wik94b}). We adopt
$q_o=0.5$ and $H_o = 70$ km s$^{-1}$ Mpc$^{-1}$ in this paper.
Table~\ref{tbl-1} gives the position, redshift, and velocity
width obtained from the optical emission lines (Koo et al. \cite{koo95}), as
well as
the integration time, the rms noise for each line, and the CO
integrated intensity and CO luminosity calculated from the CO J=3-2
upper limit.

\section{Discussion}

Our upper limits to the CO flux are comparable to the best
upper limits  in the literature for moderate to high redshift
objects. For example,
the detections of CO J=3-2 emission at high redshift are
6.7 Jy km s$^{-1}$ for IRAS F10214+4724 (Radford et al. \cite{radford})
and 8.1 Jy km s$^{-1}$ for the Cloverleaf (Barvainis et al. \cite{barvainis}),
while our 3$\sigma$ upper limits range from 2 to 8
Jy km s$^{-1}$. If our galaxies had comparable
CO fluxes to IRAS F10214+4724 or the Cloverleaf quasar, we would
have detected them with our observations. In addition, if we assume
the amplification due to lensing is a factor of 10 in the two
high redshift galaxies, their CO luminosities $L_{CO}$ (converted to our
cosmology) are $8.9\times 10^9$ and $1.3\times 10^{10}$ K km s$^{-1}$ pc$^2$,
respectively. Thus, we would have detected either of these two
galaxies, unlensed, at a redshift of $z \sim 0.5$.

Since these faint blue galaxies are thought to be distant counterparts to
HII galaxies, we should also compare our upper limits with CO
observations of nearby dwarf galaxies. The CO J=1-0 luminosities
of the starburst galaxy M82 and the HII galaxy UM448 are both
$L_{CO} \sim 5\times 10^8$ K km s$^{-1}$ pc$^2$ (calculated
from Young et al. \cite{young}; Sage et al. \cite{sage}).
Unfortunately, our best
upper limits are still a factor of 4-8 larger than the luminosities
of these nearby dwarf galaxies, and so we would not have detected M82
or UM448 at $z\sim 0.5$.

For galaxies in the local universe with near-solar metallicities and
normal rates of star formation (i.e. not starburst galaxies), the
mass of molecular hydrogen gas is related to the CO luminosity in
the J=1-0 line by $M_{H_2} = 4.8 L_{CO}$ M$_\odot$ (i.e.
Solomon et al. \cite{sol87}). Since we have observed the CO J=2-1 and
J=3-2 lines, we must consider the excitation of the gas in
estimating molecular gas masses. In galactic nuclei, the three transitions
have similar strengths (Braine \& Combes \cite{braine}; G\"usten et al.
\cite{gusten}),
while in the disks of spiral galaxies, the J=2-1/1-0 ratio is typically
0.5-0.7 (Braine \& Combes \cite{braine}; Sakamoto et al. \cite{sakamoto})
and the
3-2/2-1 ratio is 0.7 (Wilson et al. \cite{wilson97}). Given the compact
and starburst nature of these systems, we will assume that the three
lowest CO line ratios have equal strengths, in which case our CO J=3-2
observations give the best upper limit to the molecular mass. If in fact
these galaxies are more like normal spirals in their excitation, our
gas masses will be underestimated by a factor of about two.

The CO-to-H$_2$ conversion factor is known
to depend on the metallicity of the galaxy (Wilson \cite{wilson95}) and is
also expected to be different in the unusual conditions present in
a starburst galaxy. The metallicities of these galaxies are estimated
to be $\sim 0.7$ solar (Guzman et al. \cite{guzman}), which implies a roughly
normal CO-to-H$_2$ conversion factor (Wilson \cite{wilson95}).
Assessing the impact
of the starburst environment is more difficult, although Solomon et al.
(\cite{sol97})
suggest that the conversion factor is roughly four times smaller
in luminous infrared galaxies than in normal spirals. To take into
account the intense star formation in these galaxies, we will
calculate the H$_2$ mass using $M_{H_2} = 1.2 L_{CO}$ M$_\odot$; note that
this may underestimate the gas masses by up to a factor of four if
the star formation environment is relatively normal.

   \begin{table}
      \caption[]{Comparison of virial and gas masses}
\label{tbl-2}
\begin{flushleft}
\begin{tabular}{lll}
\noalign{\smallskip}
\hline
\noalign{\smallskip}
Galaxy & $M_{vir}$ & $M_{H_2}$ \\
& $10^9$ M$_\odot$ & $10^9$ M$_\odot$ \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
SA57-1501 & 17 & $<$3.2 \\
SA57-7042 & 60 & $<$14 \\
SA57-10601 & 8.4 & $<$3.4 \\
SA57-17731 & 48 & $<$8.9 \\
Herc-1-13088 & 9.2 & $<$6.1 \\
\noalign{\smallskip}
\hline
\end{tabular}
\end{flushleft}
   \end{table}

Table~\ref{tbl-2} compares the upper limits to the gas masses with the virial
masses of the galaxies. The virial masses were calculated as in
Guzman et al. (\cite{guzman}), and then scaled up by a factor
of 4 to account for possible underestimate of the total mass of
the galaxy using the optical data (see discussion in Guzman et al.
\cite{guzman}). (These virial mass estimates will be even more uncertain
if these objects are predominantly disk-like rather than spheroidal
systems, as assumed in the calculation.)
Since the radii of our galaxies have not
been measured, we adopt the average effective radius $R_e = 0.96$ kpc
(converted to our cosmology) measured for seven faint blue galaxies by
Koo et al. (\cite{koo94}). Table~\ref{tbl-2}
shows that, even with our conservative assumptions,
which increase the total mass of the galaxy and decrease the gas mass,
our upper limits are consistent with the faint blue galaxies containing
between 19\% and 66\% of their total mass in the form of molecular gas.
Depending on how much gas  they can
retain  to fuel later star formation episodes, these galaxies
may either be the
progenitors of present day dwarf elliptical galaxies (if most
of the gas is lost in the current burst, Koo et al. \cite{koo95})
 or HII galaxies (if a significant fraction of their gas survives
the current burst).
Unfortunately, these observations provide no constraint on the interesting
question of whether or not these galaxies still contain large quantities
of molecular gas.

\section{Conclusions}

We have obtained sensitive upper limits on the CO J=2-1 and CO J=3-2
emission lines for five faint blue galaxies with redshifts $z\sim 0.5$
for which accurate redshifts and linewidths had been determined
previously from optical observations. Our upper limits
are comparable to the best upper limits published for
moderate to high redshift objects in the literature, and are sensitive
enough to have detected the
luminous infrared galaxy IRAS F10214+4724 if it were located at this
redshift and unlensed. However, our sensitivity is insufficient
to detect either the
prototype starburst galaxy M82 or the HII galaxy UM448 if they were
located at this redshift. Our upper limits for the CO emission are
consistent with these galaxies containing between 19\% and 66\% of their
 total mass in the form of molecular hydrogen. Since whether
these galaxies will evolve into dwarf ellipticals or retain enough
gas to burst again as HII galaxies depends on their ability to
retain gas during the observed burst of star formation, our observations
are unable to provide any useful information on the
ultimate fate of these galaxies.

\begin{acknowledgements}
The research of C.D.W. is supported by a grant from NSERC (Canada).
\end{acknowledgements}


The emergence of number and syntax units in LSTM language models

Abstract: Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long-distance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long-distance number information is largely managed by two `number units'. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.

\section*{Acknowledgments}
We would like to thank Kristina Gulordava, Jean-Remi King, Tal Linzen, Gabriella Vigliocco and Christophe Pallier for helpful feedback and comments on the work.

\section{Summary and discussion}
We provided the first  detailed description of the underlying mechanism by which an LSTM language-model performs long-distance number agreement. Strikingly, simply training an LSTM on a language-model objective on raw corpus data brought about single units carrying exceptionally specific linguistic information. Three of these units were found to form a highly interactive local network, which makes up the central part of a `neural' circuit performing long-distance number agreement.

One of these units encodes and stores grammatical number information
when the main subject of a sentence is singular, and it successfully
carries this information across long-range dependencies. Another unit
similarly encodes plurality. These number units show that a highly
local encoding of linguistic features can emerge in LSTMs during
language-model training, as was previously suggested by theoretical
studies of artificial neural networks \cite[e.g.,][]{Bowers:2009} and in
neuroscience \cite[e.g.,][]{Kutter:etal:2018}.

Our analysis also identified units whose activity correlates with syntactic complexity. These units, as a whole, affect performance on the agreement tasks. We further found that one of them encodes the main subject-verb dependency across various syntactic constructions. Moreover, the highest afferent weights to the forget and input gates of both LR-number units were from this unit. A natural interpretation is that this unit propagates syntax-based remember and update flags that control when the number units store and release information.

Finally, number is also redundantly encoded in a more distributed way, but the latter mechanism is unable to carry information across embedded syntactic structures. The computational burden of tracking number information thus gave rise to two types of units in the network,  encoding similar information  with distinct properties and dynamics.

The relationship we uncovered and characterized between syntax and number units suggests that agreement in an LSTM language-model cannot be entirely explained away by superficial heuristics, and the networks have, to some extent, learned to build and exploit structure-based syntactic representations, akin to those conjectured to support human-sentence processing.

In future work, we intend to explore how the encoding pattern we found varies across network architectures and hyperparameters, as well as across languages and domains. We also would like to investigate the timecourse of emergence of the found behaviour over training time.

More generally, we hope that our study will inspire more analyses of the inner dynamics of LSTMs and other sequence-processing networks, complementing the currently popular ``black-box probing'' approach. Besides bringing about a mechanistic understanding of language processing in artificial models, this could inform work on human-sentence processing. Indeed, our study yields particular testable predictions on brain dynamics, given that the computational burden of long-distance agreement remains the same for artificial and biological neural network, despite implementation differences and different data sizes required for language acquisition. We conjecture a similar distinction between SR and LR units to be found in the human brain, as well as an interaction between syntax-processing and feature-carrying units such as the LR units, and plan to test these in future work.

\section{Introduction}

In the last years, recurrent neural networks (RNNs), and particularly
long-short-term-memory (LSTM) architectures
\cite{Hochreiter:Schmidhuber:1997}, have been successfully applied to
a variety of NLP tasks. This has spurred interest in whether these
generic sequence-processing devices are discovering genuine structural
properties of language in their training data, or whether their
success can be explained by opportunistic surface-pattern-based
heuristics.

Until now, this debate has mostly relied on ``behavioural'' evidence:
The LSTM had been treated as a black box, and its capacities had been indirectly
inferred by its performance on linguistic tasks. In this study, we took a
complementary approach inspired by neuroscience: We thoroughly investigated
the inner dynamics of an LSTM language model performing a number agreement task,
striving to achieve a mechanistic understanding of how it accomplishes
it. We found that the LSTM had specialized two ``grandmother'' cells
\cite{Bowers:2009} to carry number features from the subject to the
verb across the intervening material.\footnote{In the neuroscientific
  literature, ``grandmother'' cells are (sets of) neurons coding for
  specific information, e.g., about your grandmother, in a
  non-distributed manner.} Interestingly, the LSTM also
possesses a more distributed mechanism to predict number when subject
and verb are close, with the grandmother number cells only playing a
crucial role in more difficult long-distance cases. Crucially, we
independently identified a set of cells tracking syntactic structure,
and found that one of them encodes the presence of an embedded phrase
separating the main subject-verb dependency, and has strong efferent
connections to the long-distance number cells, suggesting that the
network relies on genuine syntactic information to regulate
agreement-feature percolation.

Our analysis thus provides direct evidence for the claim that LSTMs
trained on unannotated corpus data, despite
lacking significant linguistic priors, learn to perform
structure-dependent linguistic operations. In turn, this suggests that
raw linguistic input and generic memory mechanisms, such as those
implemented in LSTMs, may suffice to trigger the induction of
non-trivial grammatical rules.

\section{Related work}

Starting with the seminal work of \newcite{Linzen:etal:2016}, a long-distance number agreement task has emerged as a standard way to probe the syntactic capabilities of neural language models.
In the number agreement task, a model is asked to predict the verb in a sentence where the subject and main verb are separated by one or more intervening nouns (``the \textbf{boy} near the \textit{cars} \textbf{greets}\ldots'') and evaluated based on how often it predicts the right verb form.

Following mixed initial results by Linzen and colleagues and \newcite{Bernardy:Lappin:2017}, \newcite{Gulordava:etal:2018} and
\newcite{Kuncoro:etal:2018a} have robustly established that LSTM language models
achieve near-human performance on the agreement task. While Gulordava and
colleagues provided some evidence that the LSTMs are relying on
genuine syntactic generalizations, \newcite{Kuncoro:etal:2018b} and
\newcite{Linzen:Leonard:2018} suggested that the LSTM achievements
can, at least in part, be accounted by superficial heuristics (e.g., ``percolate the number of the first noun in a sentence''). Other
recent work has extended syntax probing to other phenomena such as
negative polarity items and island constraints
\cite{Chowdhury:Zamparelli:2018,jumelet2018language,marvin2018targeted,wilcox2018rnn}.

While \newcite{Linzen:etal:2016} presented intriguing qualitative data
showing cells that track grammatical number in a network directly trained on
the agreement task, most of the following work focused on testing the
network output behaviour, rather than on understanding how the latter
follows from the inner representations of the network. Another research line
studied linguistic processing in neural networks through `diagnostic classifiers', that is, classifiers trained to
predict a certain property from network activations
\cite[e.g.,][]{gelderloos2016phonemes,Adi:etal:2017,alain2017understanding,Hupkes:etal:2017}. This approach may give insight into which information is encoded by the
network in different layers or at different time points, but it only
provides indirect evidence about the specific mechanics of linguistic
processing in the network.

Other studies are closer to our approach in that they attempt to
attribute function to specific network cells, often by means of
visualization
\cite{Karpathy:etal:2016,li2016visualizing,tang2017memory}. \newcite{Radford:etal:2017},
for example, detected a ``sentiment'' grandmother cell in a
language-model-trained network.  \newcite{Kementchedjhieva:Lopez:2018}
recently found a character-level RNN to track morpheme boundaries in a
single cell. We are however not aware of others studies systematically
characterizing the processing of a linguistic phenomenon at the level of
RNN cell dynamics, as is the attempt in the study hereby presented.

\subsection{Syntax-number units connections}

We finally look at the connections that were learned by the LSTM
between syntax unit \unit{2}{500}, which appears to be more closely involved in
tracking subject-verb agreement, and the LR number units, as well as at
the connections between the LR-number units themselves. For each unit pair, there are 4
connection types, one for each component of the target cell (to the 3
gates and to the update candidate). We focus on input and forget gates, as they control the flow and storage of number information.

Figures \ref{fig:interaction-input} and \ref{fig:interaction-forget}
show the distributions of all afferent recurrent weights to the input
and forget gates of the LR-number units, scaled by the maximal activity
$h_t$ of the pre-synaptic units during the nounPP task (this scaling evaluates
the \textit{effective} input to the units and did not change the conclusions described below). We found that the weights from the
syntax unit to the forget gate of both \unit{2}{126} and \unit{2}{338}
are exceptionally high in the positive direction compared to all other
afferent connections in the network ($z-score=8.1, 11.2$, respectively) and
those to their input gates exceptionally negative ($z-score=-16.2,
-7.2$). Since the cell activity of syntax unit \unit{2}{500} is
positive across the entire subject-verb dependency (e.g., Figure
\ref{fig:syntax-unit-double-subjrel}), the connectivity from the
syntax unit drives the number unit forget gates towards one
($W^f_{776, 1150}h^{1150}\gg0$ and $W^f_{988, 1150}h^{1150}\gg0$; $t_{subject}<t<t_{verb}$) and their input gates towards zero
($W^i_{776, 1150}h^{1150}\ll0$ and $W^i_{988, 1150}h^{1150}\ll0$). Looking at the right-hand-side of
Eq.~(\ref{eq:update-rule}), this means that the first term becomes
dominant and the second vanishes, suggesting that, across the entire
dependency, the syntax unit conveys a `remember flag' to the number
units. Similarly, when the activity of the syntax unit becomes
negative at the end of the dependency, it conveys an `update flag'.

Last, we note that the reciprocal connectivity between the two
LR-number units is always positive, to both input and forget
gates (with $|z-score|>3$ for the \unit{2}{126}-to-\unit{2}{338} direction). Since
their activity is negative throughout the subject-verb dependency
(Figures \ref{fig:singular-unit} and \ref{fig:plural-unit}), this means
that they are \textit{mutually inhibiting}, thus steering towards an
unequivocal signal about the grammatical number of the subject to the
output layer.

\section{Experiments}\label{sec:results}
To successfully perform the NA-task, the LSTM should: (1) encode and store the grammatical number of the subject; and (2) track the main subject-verb syntactic dependency. The latter information is important for identifying the time period during which subject number should be stored, output and then updated by the network. This section describes the `neural circuit' that encodes and processes this information in the LSTM.

\begin{center}
\begin{table}[t]
\centering
\begin{tabular}{|P{2.3cm}|P{0.4cm}||P{0.6cm}|P{0.6cm}|P{0.6cm}|}
\hline
    \B \multirow{2}{*}{NA task} & \B \multirow{2}{*}{C} & \multicolumn{2}{c|}{\B Ablated}& \multirow{2}{*}{\B Full} \\
    \cline{3-4}
    & & \B \textbf{\unit{2}{126}} & \B \textbf{\unit{2}{338}} & \\
\hline

Simple & S & - &  - &  \textcolor{red}{100} \\

Adv & S & - &  - &  \textcolor{red}{100} \\

2Adv & S & - &  - &  \textcolor{red}{99.9} \\

CoAdv & S & - &  \textcolor{red}{82} &  \textcolor{red}{98.7} \\

namePP & SS & - &  - &  \textcolor{red}{99.3} \\

nounPP & SS & - &  - &  \textcolor{red}{99.2} \\

nounPP & SP &  - &  \textcolor{red}{54.2} &  \textcolor{red}{87.2} \\

nounPPAdv & SS &  - &  - & \textcolor{red}{99.5} \\

nounPPAdv & SP &  - &  \textcolor{red}{54.0} & \textcolor{red}{91.2} \\

\hline
Simple & P &  - &  - &  \textcolor{blue}{100} \\

Adv & P &  - &  - &  \textcolor{blue}{99.6} \\

2Adv & P & - &  - &   \textcolor{blue}{99.3} \\

CoAdv & P &  \textcolor{blue}{79.2} &  - &   \textcolor{blue}{99.3} \\

namePP & PS & \textcolor{blue}{39.9} &  - &   \textcolor{blue}{68.9} \\

nounPP & PS &  \textcolor{blue}{48.0} & - &   \textcolor{blue}{92.0} \\

nounPP & PP &  \textcolor{blue}{78.3} & - &   \textcolor{blue}{99.0} \\

nounPPAdv & PS & \textcolor{blue}{63.7} &  - &   \textcolor{blue}{99.2} \\

nounPPAdv & PP & - &  - &   \textcolor{blue}{99.8} \\

\hline

\B Linzen & \B - &   75.3 &  - &  93.9 \\
\hline

\end{tabular}
\caption{Ablation-experiments results: Percentage accuracy in all NA-tasks. Full: non-ablated model, C: condition, S: singular, P: plural. Red: Singular subject, Blue: Plural subject. Performance reduction less than 10\% is denoted by `-'.  \label{tab:ablation-results}}
\end{table}
\end{center}

\subsection{Long-range number units}\label{subsec:ablation}
We first tested the performance of the LSTM on the Linzen's data and on the NA-tasks in Table
\ref{tab:data-sets}. Following
\newcite{Linzen:etal:2016} and later work, we computed the likelihood
that the LSTM assigns to the main verb of each sentence given the
preceding context and compared it to the likelihood it assigns to the
wrong verb inflection. Accuracy in a given condition was measured as the proportion of sentences in this condition for which the model assigned a higher likelihood to the correct verb form than to the wrong one.

Network performance is reported in Table \ref{tab:ablation-results}
(right column -- `Full'). We first note that our results on the Linzen NA-task confirm those reported in \newcite{Gulordava:etal:2018}. For the other NA-tasks, results show that some tasks and
conditions are more difficult than others. For example, performance on
the Simple (0-distance) NA-task is better than that on the Co-Adv
NA-task, which in turn is better than that of the nounPP
tasks. Second, as expected, incongruent conditions (the
number-mismatch conditions of namePP, nounPP and nounPPAdv) reduce
network performance. Third, for long-range dependencies, reliably
encoding singular subject across an interfering noun is more difficult
than a plural subject: for both nounPP and nounPPAdv, PS is easier
than SP. A possible explanation for this finding is that in English the plural form is
almost always more frequent than the singular one, as the latter only
marks third person singular, whereas the former is identical to the
infinitive and other forms. Thus, if the network reverts to unigram
probabilities, it will tend to prefer the plural.

\paragraph{Looking for number units through ablation} Number
information may be stored in the network in either a local, sparse, or
a distributed way, depending on the fraction of active units that
carry it.  We hypothesized that if the network uses a local or sparse
coding, meaning that there's a small set of units that encode number
information, then ablating these units would lead to a drastic
decrease in performance in the NA-tasks.  To test this, we ablated
each unit of the network, one at a time, by fixing its activation to zero,
and tested on the NA-tasks.

Two units were found to have exceptional effect on network performance
(Table \ref{tab:ablation-results}, \unit{2}{126} and \unit{2}{338}
columns).\footnote{Units 1-650 belong to the first layer, 651-1300 to
  the second. All units detected by our analyses come from the latter.} Ablating them reduced network performance by more than 10\%
across various conditions, and, importantly, they were the only units
whose ablation consistently brought network performance to around
chance level in the more difficult incongruent conditions of the
namePP, nounPP and nounPPAdv
tasks.

Moreover, the ablation effect depended on the grammatical number of the subject: ablating \unit{2}{126} significantly reduced
network performance only if the subject was plural (P, PS or PP conditions) and \unit{2}{338}
only if the subject was singular (S, SP or SS conditions). In what follows, we will therefore
refer to these units as the `plural' and `singular' units, respectively,
or long-range (LR) number units when referring to both. Finally, we note that although the Linzen NA-task contained mixed stimuli from many types of conditions, the plural unit was found to have a substantial effect on average on network performance. The singular unit didn't show a similar effect in this case, which highlights the importance of using carefully crafted stimuli, as in the nounPP and nounPPAdv tasks, for understanding network dynamics. Taken
together, these results suggest a highly local coding scheme of
grammatical number when processing long-range dependencies.

\paragraph{Visualizing gate and cell-state dynamics}\label{subsec:gate-dynamics}
To understand the functioning of the number units, we now look
into their gate and state dynamics during sentence processing. We
focus on the nounPP NA-task, which is the simplest NA-task that includes a
long-range dependency with an interfering noun, in both SP and PS
conditions.

Recall the standard LSTM memory update and output rules \cite{Hochreiter:Schmidhuber:1997}:
\begin{align}
    C_t &= f_t\circ C_{t-1} + i_t\circ \widetilde{C}_t \label{eq:update-rule} \\
     h_t &= o_t\circ \tanh(C_t) \label{eq:output},
\end{align}
where $f_t, i_t, o_t \in (0,1)$ are gating scalars computed by the network, and $\widetilde{C}_t \in (-1, 1)$ is an update candidate for cell value.

\begin{figure*}[ht]
    \centering
    \begin{subfigure}{\textwidth}
            \centering
            \includegraphics[width=0.4\linewidth]{Figures/legend.pdf}
    \end{subfigure}
    \bigskip
    \begin{subfigure}{0.45\textwidth}
            \centering
            \includegraphics[width=\linewidth]{Figures/nounpp_987.pdf}
            \subcaption{\unit{2}{338} (singular)}
    \label{fig:singular-unit}
    \end{subfigure}
    \begin{subfigure}{0.45\textwidth}
            \centering
            \includegraphics[width=\linewidth]{Figures/nounpp_775.pdf}
            \subcaption{\unit{2}{126} (plural)}
    \label{fig:plural-unit}
    \end{subfigure}
    \begin{subfigure}{0.45\textwidth}
            \centering
            \includegraphics[width=\linewidth]{Figures/unit-timeseries-cartoon.pdf}
            \subcaption{Prediction (singular)}
    \label{fig:cartoon}
    \end{subfigure}
    \begin{subfigure}{0.45\textwidth}
        \centering
        \includegraphics[height=4.3cm]{Figures/weight_dists_verbs.png}
        \caption{Efferent weights of the LR-units (\unit{2}{126} and \unit{2}{338}), the syntax unit (\unit{2}{500}; section \ref{sec:syntax-units}) and two arbitrary units (\unit{2}{1} and \unit{2}{650}).}
        \label{fig:output-weights}
    \end{subfigure}
    \caption{(a) to (c) -- Cell and gate activations during processing of sentences with a prepositional phrase between subject and verb. Values in (\subref{fig:singular-unit}) and (\subref{fig:plural-unit}) are averaged across all condition sentences, with error bars showing standard deviations. (\subref{fig:output-weights}) -- Efferent weights of specific units at the output layer to singular and plural verb forms.}
\end{figure*}

Consider now how a number unit may reliably encode and store subject
number across interfering nouns.  Figure~\ref{fig:cartoon} exemplifies
this for a singular unit, showing the desired gate and cell
dynamics. The four conditions are represented with separated curves -
red for singular subject, blue for plural, and dashed lines for
incongruent conditions. Gate and cell activity at time points
unrelated to solving the NA-task are masked with white, as we do not
make precise predictions for them.

The update rule of the LSTM cell has two terms
(Eq.~\ref{eq:update-rule}).\footnote{We abuse notation here, using the
  symbols denoting whole layers in equations (\ref{eq:update-rule}) and
  (\ref{eq:output}) to denote the components of single cells.} In the
first, $f_t \circ{} C_{t-1}$, the forget gate controls whether to keep
the previous cell content ($f_t=1$: perfect remembering) or forget it
($f_t=0$: complete forgetting). In the second,
$i_t\circ{} \tilde{C}_t$, the input gate controls whether the
information currently presented to the network, as encoded by
$\tilde{C}_t$, should be written onto the cell ($i_t=1$: full access)
or not ($i_t=0$). The singular unit can thus use these gates to
reliably store number information across long-range
dependencies. Specifically, the unit can (enumeration follows the same order as the panels in Figure~\ref{fig:cartoon}): (1) encode subject
number via $\tilde{C}_{t_{subject}}$ with different values for
singular and plural; (2) open the input gate \textit{only} when a
singular subject is presented ($i_{t_{subject}} = 1$ in red curves \textit{only}) and protect it from interfering nouns ($i_t=0, t_{subject}<t<t_{verb}$); (3) at the same time,
clear the cell from previously stored information
($f_{t_{subject}}=0$) and then store subject number across the entire
dependency ($f_t=1, t_{subject}<t<t_{verb}$); (4) this will result in
stable encoding of subject number in the cell $C_t$ throughout the
dependency; (5) finally, output subject number at the right moment,
when predicting the verb form ($o_{t_{verb}-1}=1$)
(Eq.~\ref{eq:output}).

Figures \ref{fig:singular-unit} and \ref{fig:plural-unit} present the actual gate and cell dynamics of the singular and plural units. Both units follow the general solution for reliable number storage described above. Note that for $\tilde{C}_t$ and $i_t$, and as a result also for $C_t$, the plural unit `mirrors' the singular unit with respect to subject number (red curves of PP and PS vs. blue curves of SS and SP). This is in accordance with the results of the ablation experiments, which showed that ablating these units had an effect that depended on the grammatical number of the subject (Table \ref{tab:ablation-results}). This provides complementary support for the identification of these units as `singular' and `plural'.

A single divergence between the solution depicted in Figure \ref{fig:cartoon} and the actual dynamics of the number units is that input gate activity is smaller, but not zero, at the time step immediately following the subject. One speculative explanation is that this might be useful to process compound nouns. In these cases, subject number information is stored with the second noun, whereas in the case of simple nouns there is no `risk' of encountering an interfering noun immediately after the subject, making the delay in closing the gate safe.

The singular and plural units had emerged at the second layer of the network. This seems appropriate since number information needs to be directly projected to the output layer for correct verb-form prediction. Moreover, number-unit output should be projected differently to singular and plural verb forms in the output layer, only increasing activity in output units representing the suitable form. For example, for the singular unit, since singular subjects are encoded with a negative value ($C_{t_{verb}-1}<-1$ in figure \ref{fig:singular-unit}), the more negative its efferent weights to singular verb forms in the output layer, the higher the probabilities of these verb forms would be. Figure \ref{fig:output-weights} shows the efferent weights of the LR-number units to all verbs in our data-sets. We found that, indeed, the efferent weights to the singular and plural verb forms are segregated from each other, with weight signs that correspond to the negative encoding of subject number used by both singular and plural units. Two other arbitrary units, \unit{2}{1} and \unit{2}{650}, and the syntax unit \unit{2}{500} to be described below (Section \ref{sec:syntax-units}) do not have segregated efferent weights to verb forms, as expected.

\subsection{Short-range number information}
\begin{figure}
    \centering
    \includegraphics[height=4cm, width=8cm]{Figures/GAT1d_cell_.png}
    \caption{Generalization across time of subject-number prediction. Error bars represent standard deviations across cross-validation splits.}
    \label{fig:GAT}
\end{figure}

Performance on the easier NA-tasks (Simple, Adv, 2Adv) was not
impaired by single-unit ablations. This suggests that number may be
encoded also elsewhere in the network, perhaps via a more distributed
code. To verify this, we tested whether subject number can be decoded from the whole
pattern of activities in the network (excluding the two LR-number units)
and whether this decoding is stable across time \cite[see][for similar
observations and related methods]{Giulianelli:etal:2018}. We expected
this distributed activity to track number in a
small time window after the subject, but, unlike the LR-number units,
to be affected by incongruent intervening nouns.

We trained a linear model to predict the grammatical number of the
subject from network activity in response to the presentation of the
subject, and tested its prediction on test sets from all time points
\cite{King:Dehaene:2014}, in incongruent conditions only of the nounPP
task. We used Area under of Curve (AUC) to evaluate model
performance. Figure \ref{fig:GAT} shows decoding across time of
subject number from cell activity of each number unit separately and
from cell activity of the entire network without these two units
(`Full model minus LR-units'). Results show that number information
can be efficiently decoded from other units in the network, and that
this information can be carried for several time steps (relatively
high AUC up to the second determiner). However, the way in which these
units encode number is sensitive to the last encountered noun, with
AUC decreasing to zero around the second noun (`cars'), whereas test
performance of the models trained on cell activity of the LR-number
units is consistently high. This confirms that number prediction is
supported both by the LR-number units, and by distributed activation
patterns of other short-range (SR) number units. The latter, however,
are not syntax-sensitive, and simply encode the number of the last
noun encountered.

\begin{figure*}[ht!]
    \begin{subfigure}{0.32\textwidth}
            \centering
            \includegraphics[width=\linewidth, height=1.9cm]{Figures/adv_conjunction_1149_cell}
            \caption{2Adv}
            \label{fig:syntax-unit-2Adv}
    \end{subfigure}
    \begin{subfigure}{0.32\textwidth}
            \centering
            \includegraphics[width=\linewidth, height=1.9cm]{Figures/nounpp_1149_cell.png}
            \caption{nounPP}
            \label{fig:syntax-unit-nounpp}
    \end{subfigure}
    \begin{subfigure}{0.32\textwidth}
            \centering
            \includegraphics[width=\linewidth, height=1.9cm]{Figures/subjrel_that_1149_cell.png}
            \caption{subject relative}
            \label{fig:syntax-unit-subjrel}
    \end{subfigure}
    \begin{subfigure}{\textwidth}
            \centering
            \includegraphics[width=\linewidth, height=2cm]{Figures/double_subjrel_that_1149_cell.png}
            \caption{Two embeddings with subject relatives}
            \label{fig:syntax-unit-double-subjrel}
    \end{subfigure}
\caption{Cell activity of syntax unit \unit{2}{500} while processing various syntactic structures. Values averaged across all stimuli in an NA-task, with error bars representing standard deviations. Relative clause NA-task stimuli were specifically generated for this visualization.}
\end{figure*}

A full description of the SR-number units is beyond our scope. However, we note that 10 SR-number units in the second layer of the
network were identified, which had efferent weights with a similar segregated
structure as that of the LR units (Figure
\ref{fig:output-weights}). These units were indeed sensitive to the last
encountered noun: subject number could be decoded from single-unit cell activity
during its presentation (AUC$>0.9$), but activity `swaps' once an
interfering noun appears (i.e., AUC decreases to zero in a
generalization-across-time analysis). Finally, to validate the role of SR-number units in encoding number for easier NA-tasks, we ablated both SR and LR number
units (12 in total) or SR units only (10 in total) and evaluated network performance on these NA-tasks. Both experiments resulted in a significant reduction in task performance compared to 1,000 random equi-size ablations ($p<0.01$ in all `easier' tasks).

Intriguingly, we observed qualitatively that LR units are almost
always making the right prediction, even when the network predicts the
wrong number. The wrong outcome, in such cases, might be due to
interference from the syntax-insensitive SR units. We leave the study of LR-SR
unit interplay to future work.

\subsection{Syntax units}
\label{sec:syntax-units}

\begin{figure*}[ht]
    \centering
    \begin{subfigure}{0.49\textwidth}
            \centering
            \includegraphics[height=3.3cm,width=\textwidth]{Figures/gate_Input_afferent_interactions.png}
            \caption{Input gate}
            \label{fig:interaction-input}
    \end{subfigure}
    \begin{subfigure}{0.49\textwidth}
           \centering
          \includegraphics[height=3.3cm,width=\textwidth]{Figures/gate_Forget_afferent_interactions.png}
          \caption{Forget gate}
          \label{fig:interaction-forget}
    \end{subfigure}
    \caption{Connectivity among the syntax unit \unit{2}{500} and
      LR-number units \unit{2}{126} and \unit{2}{338}. Projecting
      units are on the table rows. Blue background highlights outlier
      values ($|z-score|>3$). Weights from the syntax unit are marked in red
      and are explicitly labeled in the plots, which show the overall distributions of afferent weights to each number unit.}
\label{fig:interaction}
\end{figure*}

We saw how the input and forget gates of the LR-number units control the flow of subject-number information. It remains unclear, however, how the dynamics of these gates are controlled by the network. We hypothesized that other units in the network may encode information about the syntactic structure of the sentence, and thus about the subject-verb dependency. These units could then control and coordinate the opening and closing of the input and forget gates of the number units.

To identify such 'syntax' units, we tested from which units syntactic information can be efficiently decoded. We used depth of the syntactic tree as a proxy for syntactic structure \cite{Nelson:etal:2017} and trained an L2-regularized regression model to predict syntactic tree-depth from the hidden-state activity of all units. In all experiments, we used the data presented in Section \ref{sec:the_data} above and performed a nested 5-fold cross-validation procedure. Word frequency, which was added as a covariate to the model, had a negligible effect on the results. Syntactic tree-depth was found to be efficiently decodable from network activity ($R^2_{test-set}=0.85\pm0.009$; covariate-corrected). A small subset of `syntax' units had relatively high weights in the regression model (mean weight = $7.6\times{}10^{-4}$, SD=$7.86\times{}10^{-2}$; cutoff for outlier weights was set to three SDs). Since the interpretation of the regression weights may depend on possible correlations among the features, we also tested the causal effect of these units on NA-task performance. Ablating the syntax units together resulted in significant performance reduction in NA-tasks that have an interfering noun: Linzen NA-task: $p=0.024$, nounPPAdv-SP: $p=0.011$, nounPPAdv-PS: $p=0.034$, nounPP-SP: $p<0.001$ and marginally significant in nounPP-PS: $p=0.052$ (compared to 1000 random ablations of subsets of units of the same size).

To gain further insight regarding the functioning of the syntax units, we next visualized their gate and cell dynamics during sentence processing. We found that cell activity of unit \unit{2}{500}, which also had one of the highest weights in the regression model, was remarkably structured. The activity of
this unit increases across the entire subject-verb dependency and drops abruptly right after. Figures
\ref{fig:syntax-unit-2Adv} and \ref{fig:syntax-unit-nounpp} show cell activity of this unit during the processing of stimuli from the 2Adv and nounPP tasks. We found the same dynamics in cases where another
verb occurs between subject and main verb, as in subject relatives (Figure \ref{fig:syntax-unit-subjrel}), and in exceptionally long-distance dependencies with two interfering nouns and verbs (Figure \ref{fig:syntax-unit-double-subjrel}). Taken together, these results suggest that unit \unit{2}{500} consistently encodes subject-verb dependencies in a syntax-sensitive manner. Other syntax units did not show an easily interpretable dynamics and had no clear interactions with the number units in the analysis discussed next. This suggests that they perform different syntactic, or possibly other, functions.

\section{Setup}\label{sec:the_data}

\paragraph{Language model}\label{ssec:lstm_lm}
We study the pretrained LSTM language model made available by
\newcite{Gulordava:etal:2018}. This model is composed of a
650-dimensional embedding layer, two 650-dimensional hidden layers,
and an output layer with vocabulary size 50,000.
The model was trained on Wikipedia data, without fine-tuning for number
agreement, and obtained perplexity close to state of the art in the experiments of Gulordava et al.\footnote{Key findings reported below were also replicated with the same model trained with different initialization seeds and variations with different hyper-parameters.}

\paragraph{Number-agreement tasks}
\begin{table}[tb]
  \centering
  \begin{footnotesize}
  \begin{tabular}{l@{\hskip1pt}l}
    \B Simple & the \textbf{boy} \textbf{greets} the guy\\
    \B Adv & the \textbf{boy} probably \textbf{greets} the guy\\
    \B 2Adv & the \textbf{boy} most probably \textbf{greets} the guy\\
    \B CoAdv &  the \textbf{boy} openly and deliberately \textbf{greets} the guy\\
    \B NamePP & the \textbf{boy} near Pat \textbf{greets} the guy\\
    \B NounPP & the \textbf{boy} near the car \textbf{greets} the guy\\
    \B NounPPAdv & the \textbf{boy} near the car kindly \textbf{greets} the guy\\
  \end{tabular}
  \end{footnotesize}
  \caption{NA tasks illustrated by representative
    singular sentences.}
  \label{tab:data-sets}
\end{table}

We complement analysis of the naturalistic, corpus-derived
number-agreement test set of \newcite{Linzen:etal:2016}, in the
version made available by \newcite{Gulordava:etal:2018}, with
synthetically generated data-sets. Each synthetic number-agreement
task (NA-task) instantiates a fixed syntactic structure with varied
lexical material, in order to probe subject-verb number agreement in
controlled and increasingly challenging setups.\footnote{We exclude,
  for the time being, agreement across a relative clause, as it comes
  with the further complication of accounting for the extra agreement
  process taking place inside the relative clause.} The different
structures are illustrated in Table \ref{tab:data-sets}, where all forms are in the singular. Distinct sentences were randomly
generated by selecting words from pools of 20
subject/object nouns, 15 verbs, 10 adverbs, 5 prepositions, 10 proper
nouns and 10 location nouns. The items were selected so that their
combination would not lead to semantic anomalies. For each NA-task, we
generated singular and plural versions of each sentence. We refer to
each such version as a \textit{condition}. For NA-tasks that have
other nouns occurring between subject and main verb, we also
systematically vary their number, resulting in two \textit{congruent}
and two \textit{incongruent} conditions. For example, the NounPP
sentence in the table illustrates the congruent SS (singular-singular)
condition and the corresponding sentence in the incongruent PS
(plural-singular) condition is: ``the \emph{boys} near the \emph{car}
\emph{greet} the guy''. For all NA-tasks, each condition consisted of
600 sentences

\paragraph{Syntactic depth data-set} We probed the implicit
syntax-parsing abilities of the model by testing whether its representations
predict the syntactic depth of the words they process. Following
\newcite{Nelson:etal:2017}, this was operationalized as predicting the
number of open syntactic nodes at each word, given the canonical
syntactic parse of a sentence.  We generated a data-set of sentences with
unambiguous but varied syntactic structures and annotated them with the number of open nodes at each word. For example:
``Ten$_1$ really$_2$ ecstatic$_3$ cousins$_3$ of$_4$ four$_5$ teachers$_6$
are$_2$ quickly$_3$ laughing$_4$", where indexes show the corresponding
number of open nodes.
Since syntactic depth is
naturally correlated with the position of a word in a sentence, we
used a data-point sampling strategy to de-correlate these factors. For
each length between 2 and 25 words, we randomly generated 300
sentences. From this set, we randomly picked examples uniformly
covering all possible position-depth combinations within the 7-12
position and 3-8 depth ranges. The final data-set contains 4,033
positions from 1,303 sentences.\footnote{All our data-sets are available at: \url{https://github.com/FAIRNS/Number_and_syntax_units_in_LSTM_LMs}.}