FlipDial: A Generative Model for Two-Way Visual Dialogue

We present FlipDial, a generative model for visual dialogue that simultaneously plays the role of both participants in a visually-grounded dialogue. Given context in the form of an image and an associated caption summarising the contents of the image, FlipDial learns both to answer questions and put forward questions, capable of generating entire sequences of dialogue (question-answer pairs) which are diverse and relevant to the image. To do this, FlipDial relies on a simple but surprisingly powerful idea: it uses convolutional neural networks (CNNs) to encode entire dialogues directly, implicitly capturing dialogue context, and conditional VAEs to learn the generative model. FlipDial outperforms the state-of-the-art model in the sequential answering task (one-way visual dialogue) on the VisDial dataset by 5 points in Mean Rank using the generated answers. We are the first to extend this paradigm to full two-way visual dialogue, where our model is capable of generating both questions and answers in sequence based on a visual input, for which we propose a set of novel evaluation measures and metrics.

Introduction
A fundamental characteristic of a good human-computer interaction (HCI) system is its ability to effectively acquire and disseminate knowledge about the tasks and environments in which it is involved. A particular subclass of such systems, natural-language-driven conversational agents such as Alexa and Siri, have seen great success in a number of welldefined language-driven tasks. Even such widely adopted systems suffer, however, when exposed to less circumscribed, more free-form situations. Ultimately, an implicit requirement for the wide-scale success of such systems is the effective understanding of the environments and goals of the user -an exceedingly difficult problem in the general case as it involves getting to grips with a variety of sub-problems (semantics, grounding, long-range dependencies) each of which are extremely difficult problems in themselves. One avenue to ameliorate such issues is the incorporation of visual context to help explicitly ground the language used -providing a domain in which knowledge can be anchored and extracted from. Conversely, this also provides a way in which language can be used to characterise visual information in richer terms, for example with sentences describing salient features in the image (referred to as "captioning") [13,15].
In recent years, there has been considerable interest in visually-guided language generation in the form of visual question-answering (VQA) [1] and subsequently visual dialogue [6], both involving the task of answering questions in the context of an image. In the particular case of visual dialogue, along with the image, previously seen questions and answers (i.e. the dialogue history) are also accepted, and a relevant answer at the current time produced. We refer to this one-sided or answer-only form of visual dialogue as one-way visual dialogue (1VD). Inspired by these models and aiming to extend their capabilities, we establish the task of two-way visual dialogue (2VD) whereby an agent must be capable of acting as both the questioner and the answerer.
Our motivation for this is simple -AI agents need to be able to both ask questions and answer them, often interchangeably, rather do either one exclusively. For example, a vision-based home-assistant (e.g. Amazon's Alexa) may need to ask questions based on her visual input ("There is no toilet paper left. Would you like me to order more?") but may also need to answer questions asked by humans ("Did you order the two-ply toilet paper?"). The same questionanswer capability is true for other applications. For example, with aids for the visually-impaired, a user may need the answer to "Where is the tea and kettle?", but the system may equally need to query "Are you looking for an Earl Grey or Rooibos teabag?" to resolve potential ambiguities.
We take one step toward this broad research goal with FLIPDIAL, a generative model capable of both 1VD and 2VD. The generative aspect of our model is served by using the conditional variational auto-encoder (CVAE), a framework for learning deep conditional generative models while simultaneously amortising the cost of inference in such models over the dataset [17,24]. Furthermore, inspired by the recent success of convolutional neural networks (CNNs) in language generation and prediction tasks [11,14,21], we explore the use of CNNs on sequences of sequences (i.e. a dialogue) to implicitly capture all sequential dependences through the model. Demonstrating the surprising effectiveness of this approach, we show sets of sensible and diverse answer generations for the 1VD task in Fig. 1.
We here provide a brief treatment of works related to visual dialogue. We reserve a thorough comparison to Das et.al. [6] for §4.3, noting here that our fully-generative convolutional extension of their model outperforms their stateof-the-art results on the answering of sequential visual-based questions (1VD). In another work, Das et.al. [7] present a Reinforcement Learning based model to do 1VD, where they instantiate two separate agents, one each for questioning and answering. Crucially, the two agents are given different information -with one (QBot) given the caption, and the other (ABot) given the image. While this sets up the interesting task of performing image retrieval from natural-language descriptions, it is also fundamentally different from having a single agent perform both roles. Jain et.al. [12] explore a complementary task to VQA [1] where the goal is instead to generate a (diverse) set of relevant questions given an image. In their case, however, there is no dependence on a history of questions and answers. Finally, we note that Zhao et.al. [27] employ a similar model structure to ours, using a CVAE to model dialogue, but condition their model on discourse-based constraints for a purely linguistic (rather than visuo-linguistic) dataset. The tasks we target, our architectural differences (CNNs), and the dataset and metrics we employ are distinct. Our primary contributions in this work are therefore: • A fully-generative, convolutional framework for visual dialogue that outperforms state-of-the-art models on sequential question answering (1VD) using the generated answers, and establishes a baseline in the challenging twoway visual dialogue task (2VD). • Evaluation using the predicted (not ground-truth) dialogue -essential for real-world conversational agents. • Novel evaluation metrics for generative models of twoway visual dialogue to quantify answer-generation quality, question relevance, and the models's generative capacity.

Preliminaries
Here we present a brief treatment of the preliminaries for deep generative models -a conglomerate of deep neural networks and generative models. In particular, we discuss the variational auto-encoder (VAE) [17] which given a dataset X with elements x ∈ X , simultaneously learns i) a variational approximation q φ (z | x) 1 to the unknown posterior distribution p θ (z | x) for latent variable z, and ii) a generative model p θ (x, z) over data and latent variables. These are both highly attractive prospects as the ability to approximate the posterior distribution helps amortise inference for any given data point x over the entire dataset X , and learning a generative model helps effectively capture the underlying abstractions in the data. Learning in this model is achieved through a unified objective, involving the marginal likelihood (or evidence) of the data, namely: The unknown true posterior p θ (z | x) in the first Kullback-Leibler (KL) divergence is intractable to compute making the objective difficult to optimise directly. Rather a lower-bound of the marginal log-likelihood log p θ (x), referred to as the evidence lower bound (ELBO), is maximised instead. By introducing a condition variable y, we capture a conditional posterior approximation q φ (z | x, y) and a conditional generative model p θ (x, z | y), thus deriving the CVAE [24]. Similar to Eq. (1), the conditional ELBO is: where the first term is referred to as the reconstruction or negative cross entropy (CE) term, and the second, the regularisation or KL divergence term. Here too, similar to the VAE, q φ (z | x, y) and p θ (z | y) are typically taken to be isotropic multivariate Gaussian distributions, whose parameters (µ q , σ 2 q ) and (µ p , σ 2 p ) are provided by deep neural networks (DNNs) with parameters φ and θ, respectively. The generative model likelihood p θ (x | z, y), whose form varies depending on the data type -Gaussian or Laplace for images and Categorical for language models -is also parametrised similarly. In this work, we employ the CVAE model for the task of eliciting dialogue given contextual information from vision (images) and language (captions).

Generative Models for Visual Dialogue
In applying deep generative models to visual dialogue, we begin by characterising a preliminary step toward it, VQA. In VQA, the goal is to answer a single question in the context of a visual cue, typically an image. The primary goal for such a model is to ensure that the elicited answer conforms to a stronger notion of relevance than simply answering the given question -it must also relate to the visual cue provided. This notion can be extended to one-way visual dialogue (1VD) which we define as the task of answering a sequence of questions contextualised by an image (and a short caption describing its contents), similar to [6]. Being able to exclusively answer questions, however, is not fully encompassing of true conversational agents. We therefore extend 1VD to the more general and realistic task of two-way visual dialogue (2VD). Here the model must elicit not just answers given questions, but questions given answers as well -generating both components of a dialogue, contextualised by the given image and caption. Generative 1VD and 2VD models introduce stochasticity in the latent representations.
As such, we begin by characterising our generative approach to 2VD using a CVAE. For a given image i and associated caption c, we define a dialogue as a sequence of question-answer pairs d 1:T = (q t , a t ) T t=1 , simply denoted d when sequence indexing is unnecessary. Additionally, we denote a dialogue context h. When indexed by step as h t , it captures the dialogue subsequence d 1:t .
With this formalisation, we characterise a generative model for 2VD under latent variable z as p θ (d, with the corresponding recognition model defined as q φ (z | d, i, c, h). Note that with relation to Eq. (2), data x is dialogue d and the condition variable is y = {i, c, h}, giving: with the graphical model structures shown in Fig. 2.  3) is general enough to be applied to single question-answering (VQA) all the way to full twoway dialogue generation (2VD). Taking a step back from generative 2VD, we can re-frame the formulation for generative 1VD (i.e. sequential answer generation) by considering the generated component to be the answer to a particular question at step t, given context from the image, caption and the sequence of previous question-answers. Simply put, this corresponds to the data x being the answer a t , conditioned on the image, its caption, the dialogue history to t-1, and the current question, or y = {i, c, h t−1 , q t }. For simplicity, we denote a compound context as h + t = h t−1 , q t and reformulate Eq. (3) for 1VD as: with the graphical model structures shown in Fig. 3. Our baseline [6] for the 1VD model can also be represented in our formulation by taking the variational posterior and generative prior to be conditional Dirac-Delta distributions. That is, . This transforms the objective from Eq. (4) by a) replacing the expectation of the log-likelihood over the recognition model by an evaluation of the log-likelihood for a single encoding (one that satisfies the Dirac-Delta), and b) ignoring the D KL regulariser, which is trivially 0. This computes the marginal likelihood directly as just the model Note that while such models can "generate" answers to questions by sampling from the likelihood function, we typically don't call them generative since they effectively make the encoding of the data and conditions fully deterministic. We explore and demonstrate the benefit of a fully generative treatment of 1VD in §4.3. It also follows trivially that the basic VQA model (for single question-answering) itself can be obtained from this 1VD model by simply assuming there is no dialogue history (i.e. step length T = 1).

"Colouring" Visual Dialogue with Convolutions
FLIPDIAL's convolutional formulation allows us to implicitly capture the sequential nature of sentences and sequences of sentences. Here we introduce how we encode questions, answers, and whole dialogues with CNNs.
We begin by noting the prevalence of recurrent approaches (e.g. LSTM [10], GRU [5]) in modelling both visual dialogue and general dialogue to date [6,7,8,12,27]. Typically recurrence is employed at two levels -at the lower level to sequentially generate the words of a sentence (a question or answer in the case of dialogue), and at a higher level to sequence these sentences together into a dialogue.
Recently however, there has been considerable interest in convolutional models of language [3,11,14,21], which have shown to perform at least as well as recurrent models, if not better, on a number of different tasks. They are also computationally more efficient, and typically suffer less from issues relating to exploding or vanishing gradients for which recurrent networks are known [19].
In modelling sentences with convolutions, the tokens (words) of the sentence are transformed into a stack of fixed-dimensional embeddings (e.g. using word2vec [18] or Glove [20], or those learned for a specific task). For a given sentence, say question q t , this results in an embeddingq t ∈ R E×L for embedding size E and sentence length L, where L can be bounded by the maximum sentence length in the corpus, with padding tokens employed where required. This two-dimensional stack is essentially a single-channel 'image' on which convolutions can be applied in the standard manner in order to encode the entire sentence. Note this similarly applies to the answer a t and caption c, producing embeddedå t andc, respectively.
We then extend this idea of viewing sentences as 'images' to whole dialogues, producing a multi-channel language embedding. Here, the sequence of sentences itself can be seen as a stack of (a stack of) word embeddingsd ∈ R E×L×2T , where now the number of channels accounts for the num-ber of questions and answers in the dialogue. We refer to this process as "colouring" dialogue, by analogy to the most common meaning given to image channels -colour.
Our primary motivation for adopting a convolutional approach here is to explore its efficacy in extending from simpler language tasks [11,14] to full visual dialogue. We hence instantiate the following models for 1VD and 2VD: Answer [1VD]: We employ the CVAE formulation from Eq. (4) and Fig. 3 to iteratively generate answers, conditioned on the image, caption and current dialogue history. Block [1VD, 2VD]: Using the CVAE formulation from Eq. (3) and Fig. 2 we generate entire blocks of dialogue directly (i.e. h = ∅ since dialogue context is implicit rather than explicit). We allow the convolutional model to implicitly supply the context instead. We consider this 2VD, although this block architecture can also generate iteratively, and can be evaluated on 1VD (see §4.2). Block Auto-Regressive [1VD, 2VD]: We introduce an auto-regressive component to our generative model in the same sense as recent auto-regressive generative models for images [9,25]. We augment the Block model by feeding its output through an auto-regressive (AR) module which explicitly enforces sequentiality in the generation of the dialogue blocks. This effectively factorises the likelihood in Eq.
where N is the number of AR layers, and d 1 is the (intermediate) output from the standard Block model. Note, again h = ∅, and d n refers to an entire dialogue at the n-th AR layer (rather than the t-th dialogue exchange as is denoted by d t ).

Experiments
We present an extensive quantitative and qualitative analysis of our models' performance in both 1VD, which requires answering a sequence of image-contextualised questions, and full 2VD, where both questions and answers must be generated given a specific visual context. Our proposed generative models are denoted as follows: Aanswer architecture for 1VD A is a generative convolutional extension of our baseline [6] and is used to validate our methods against a standard benchmark in the 1VD task. B and B AR , like A, are generative, but are extensions capable of doing full dialogue generation, a much more difficult task. Importantly, B and B AR are flexible in that despite being trained to generate a block of questions and answers (h = ∅), they can be evaluated iteratively for both 1VD and 2VD (see §4.2). We summarise the data and condition variables for all models in Tab. 1. To evaluate performance on both tasks, we propose novel evaluation metrics which augment those of our baseline [6]. To the best of our knowledge, we are the first to report models Table 1: Data (x) and condition (y) variables for models A and B/B AR for 1VD and 2VD. Models B/B AR can be evaluated as a block or iteratively (see §4.2), accepting groundtruth (q/a) or predicted (q/â) dialogue history (see Tab. 2).

Task Model
Train Evaluate Eval method that can generate both questions and answers given an image and caption, a necessary step toward a truly conversational agent. Our key results are: • We set state-of-the-art results in the 1VD task on the Vis-Dial dataset, improving the mean rank of the generated answers by 5.66 (Tab. 3, S w2v ) compared to Das et al. [6]. • Our block models are able to generate both questions and answers, a more difficult but more realistic task (2VD). • Since our models are generative, we are able to show highly diverse and plausible question and answer generations based on the provided visual context.

Datasets:
We use the VisDial [6] dataset (v0.9) which contains Microsoft COCO images each paired with a caption and a dialogue of 10 question-answer pairs. The train/test split is 82, 783/40, 504 images, respectively. Baseline: Das et al. [6]'s best model, MN-QIH-G, is a recurrent encoder-decoder architecture which encodes the image i, the current question q t and the attention-weighted ground truth dialogue history d 1:t−1 . The output conditional likelihood distribution is then used to (token-wise) predict an answer. Our A model is a generative and convolutional extension, evaluated using existing ranking-based metrics [6] on the generated and candidate answers. We also (iteratively) evaluate our B/B AR for 1VD as detailed in §4.2 (see Tab. 3).

Network architectures and training
Following the CVAE formulation ( §3) and its convolutional interpretation ( §3.1), all our models (A, B and B AR ) have three core components: an encoder network, a prior network and a decoder network. Fig. 4 (top) shows the encoder and prior networks, and Fig. 4 (middle, bottom) show the standard and auto-regressive decoder networks.

Prior network
The prior neural network, parametrised by θ, takes as input the image i, the caption c and the dialogue context. Referring to Table 1  each word through a learnable word embedding module and stack these embeddings as described in §3.1. We encode these condition variables convolutionally to obtain y, and pass this through a convolutional block to obtain µ p and log σ 2 p , the parameters of the conditional prior p θ (z | y).

Encoder network
The encoder network, parametrised by φ, takes x and the encoded condition y (obtained from the prior network) as input. For model A, x = a t while for B/B AR , x = d = (q t , a t ) T t=1 . In all models, x is transformed through a word-embedding module into a singlechannel answer 'image' for A, or a multi-channel image of alternating questions and answers for B/B AR . The embedded output is then combined with y to obtain µ q and log σ 2 q , the parameters of the conditional latent posterior q φ (z | x, y).

Decoder network
The decoder network takes as input a latent z and the encoded condition y. The sample is transpose-convolved, combined with y and further transformed to obtain an intermediate output volume of dimension E ×L×M , where E is the word embedding dimension, L is the maximum sentence length and M is the number of dialogue entries in x (M = 1 for A, M = 2T for B variants). Following this, A and B employ a standard linear layer, projecting the E dimension to the vocabulary size V (Fig. 4 (middle)), whereas B AR employs an autoregressive module followed by this standard linear layer (Fig. 4 (bottom)). At train time, the V -dimensional output is softmaxed and the CE term of the ELBO computed. At test time, the Under each condition, the input dialogue block is filled with ground-truth or predicted history (q/a orq/â, respectively), while future entries are filled with the PAD token.
argmax of the output provides the predicted word index. The weights of the encoder and prior's learnable word embedding module and the decoder's final linear layer are shared.
Autoregressive module Inspired by PixelCNN [26] which sequentially predicts image pixels, and similar to [9], we apply N = {8, 10} size-preserving autoregressive layers to the intermediate output of model B (size E ×L×2T ), and then project E to vocabulary size V . Each layer employs masked convolutions, considering only 'past' embeddings, sequentially predicting 2T * L embeddings of size E, enforcing sequentiality at both the sentence-and dialogue-level.

KL annealing
Motivated by [4] in learning continuous latent embedding spaces for language, we employ KL annealing in the loss objectives of Eq. (3) and Eq. (4). We weight the KL term by α ∈ [0, 1] linearly interpolated over 100 epochs, and then train for a further 50 epochs (α = 1).

Network and training hyper-parameters
In embedding sentences, we pad to a maximum sequence length of L = 64 and use a word-embedding dimension of E = 256 (for word2vec, E = 300). After pre-processing and filtering the vocabulary size is V = 9710 (see supplement for further details). We use the Adam optimiser [16] with default parameters, a latent dimensionality of 512 and employ batch normalisation with momentum= 0.001 and learnable parameters. For model A we use a batch size of 200, and 40 for B/B AR . We implement our pipeline using PYTORCH [22].

Evaluation methods for block models
Although B/B AR generate whole blocks of dialogue directly (h = ∅), they can be evaluated iteratively, lending them to both 1VD and 2VD (see supplement for descriptions of generation/reconstruction pipelines). . This is 2VD since the model must generate a coherent block of both questions and answers. • Iterative evaluation. The reconstruction pipeline can generate dialogue items iteratively. At time t, the input dialogue block is filled with zeros (PAD token) and the ground-truth/predicted dialogue history to < t is slotted in (see below and Tab. 2). This future-padded block is then . At time t, the input dialogue block is filled with the history of ground-truth questions and answers up to t-1, along with the current ground-truth question. All future entries are padded -equivalent to [6] using the ground-truth dialogue history.
. Similar to d-qa, except that the input block is filled with the history of ground-truth questions and previously predicted answers along with the current ground-truth question. This is a more realistic 1VD.

Evaluation and Analysis
We evaluate our A, B, and B AR models on the 1VD and 2VD tasks. Under 1VD, we predict an answer with each time step, given an image, caption and the current dialogue history ( §4.3.1 and Tab. 3), while under 2VD, we predict both questions and answers ( §4.3.2 and Tab. 4). All three models are able to perform the first task , while only B and B AR are capable of the second task.

One-Way Visual Dialogue (1VD) task
We evaluate the performance of A and B/B AR on 1VD using the candidate ranking metric of [6] as well as an extension of this which assesses the generated answer quality (Tab. 3). Fig. 1 and Fig. 5   To compare against their baseline, we rank the 100 candidates answers by estimates of their marginal likelihood from A. This can be done with i) the conditional ELBO (Eq. (4)), and by ii) likelihood weighting (LW) in the conditional gener- Ranking by both these approaches is shown in the S M section of Tab. 3, indicating that we are comparable to the state of the art in discriminative models of sequential VQA [6,7].
Candidate ranking by word2vec cosine distance [S w2v ] The evaluation protocol of [6] scores and ranks a given set of candidate answers, without being a function of the actual answer predicted by the model,â t . This results in the rank of the ground-truth answer candidate reflecting its score under the model relative to the rest of the candidates' scores, rather than capturing the quality of the answer output by the model, which is left unobserved. To remedy this, we instead score each candidate by the cosine distance between the word2vec embedding of the predicted answerâ t and that candidate's word2vec embedding. We take the embedding of a sentence to be the average embedding over word tokens following Arora et al. [2]. In addition to accounting for the predicted answer, this method also allows semantic similarities to be captured such that if the predicted answer is similar (in meaning and/or words generated) to the ground-truth candidate answer, then the cosine distance will be small, and hence the ground-truth candidate's rank closer to 1.
We report these numbers for A, iteratively-evaluated B/B AR , and also our baseline model MN-QIH-G [6], which we re-evaluate using the word2vec cosine distance ranking (see S w2v in Tab. 3). In the case of A (GEN), we evaluate answer generations from A whereby we condition on i, c and h + t via the prior network, sample z ∼ N (z; µ p , σ 2 p ) and generate an answer via the decoder network. Here we show an improvement of 5.66 points in MR over the baseline. On the other hand, A (RECON) evaluates answer reconstructions in which z is sampled from N (z; µ q , σ 2 q ) (where groundtruth answer a t is provided). We include A (RECON) merely as an "oracle" autoencoder, observing its good ranking performance, but do not explicitly compare against it.
We also note that the ranking scores of the block models are worse (by 3-4 MR points) than those of A. This is expected since A is explicitly trained for 1VD which is not the case for B/B AR . Despite this, the performance gap between A (GEN) and B/B AR (with d-qa) is not large, bolstering our iterative evaluation method for the block architectures. Note finally that the B/B AR models perform better under d-qa than under d-qâ (by 2-3 MR points). This is also expected as answering is easier with access to the ground-truth dialogue history rather than when only the previously predicted answers (and ground-truth questions) are provided.

Two-way Visual Dialogue (2VD) task
Our flexible CVAE formulation for visual dialogue allows us to move from 1VD to the generation of both questions and answers (2VD). Despite this being inherently more challenging, B/B AR are able to generate diverse sets of questions and answers contextualised by the given image and caption. Fig. 6 shows snippets of our two-way dialogue generations.
In evaluating our models for 2VD, the candidate ranking protocol of [6] which relies on a given question to rank the answer candidates, is no longer usable when the questions themselves are being generated. This is the case for B/B AR block evaluation, which has no access to the ground-truth dialogue history, and the d-qâ iterative evaluation, when the full predicted history of questions and answers is provided (Tab. 2). We therefore look directly to the CE and KL terms of the ELBO as well as propose two new metrics, sim c,q and sim , to compare our methods in the 2VD task: • Question relevance (sim c,q ). We expect a generated question to query an aspect of the image, and we use the presence of semantically similar words in both the question and image caption as a proxy of this. We compute the cosine distance between the (average) word2vec embedding of each predicted question q t and that of the caption  c, and average over all T questions in the dialogue (closer to 1 indicates higher semantic similarity). • Latent dialogue dispersion (sim ). For a generated dialogue block d g , sim computes the KL divergence D KL (q φ (z|d g , i, c) q φ (z|d, i, c)), measuring how close the generated dialogue is to the true dialogue d in the latent space, given the same image i and caption c. From Tab. 4, we observe a decrease in the loss terms as the auto-regressive capacity of the model increases (none → 8 → 10), suggesting that explicitly enforcing sequentiality in the dialogue generations is useful. For sim within a particular model, the dispersion values are typically larger for the harder task (without dialogue context). We also observe that dispersion increases with number of AR layers, suggesting AR improves the diversity of the model outputs, and avoids simply recovering data observed at train time. While the proposed metrics provide a novel means to evaluate dialogue in a generative framework, like all languagebased metrics, they are not complete. The question-relevance metric, sim c,q , can stagnate, and neither metric precludes redundant or nonsensical questions. We intend for these metrics to augment the bank of metrics available to evaluate dialogue and language models. Further evaluation, including i) using auxiliary tasks, as in the image-retrieval task of [7], to drive and evaluate the dialogues, and ii) turning to human evaluators to rate the generated dialogues, can be instructive in painting a more complete picture of our models.

Conclusion
In this work we propose FLIPDIAL, a generative convolutional model for visual dialogue which is able to generate answers (1VD) as well as generate both questions and answers (2VD) based on a visual context. In the 1VD task, we set new state-of-the-art results with the answers generated by our model, and in the 2VD task, we are the first to establish a baseline, proposing two novel metrics to assess the quality of the generated dialogues. In addition, we propose and evaluate our models under a much more realistic setting for both visual dialogue tasks in which the predicted rather than ground-truth dialogue history is provided at test time. This challenging setting is more akin to real-world situations in which dialogue agents must be able to evolve with their predicted exchanges. We emphasize that research focus must be directed here in the future. Finally, under all cases, the sets of questions and answers generated by our models are qualitatively good: diverse and plausible given the visual context. Looking forward, we are interested in exploring additional methods for enforcing diversity in the generated questions and answers, as well as extending this work to explore recursive models of reasoning for visual dialogue.

B. Extended Quantitative Results on 1VD task
Tab. 3 in the main paper evaluates A and B/B AR in the task of one-way visual dialogue (1VD). Here we shed light on these numbers and the metrics used to obtain them. We also present a more extensive quantitative analysis of B/B AR in the 1VD task (see Tab. 5).

Evaluating B/B AR on 1VD
We extend Tab. 3 with-Tab. 5, which further compares B/B AR under the iterative evaluation settings of d-qa and d-qâ, using the CE and KL terms of the ELBO and our two new metrics, sim c,q and sim . We observe that B/B AR (d-qa) shows superior performance of around 7-10 points in MR over B/B AR (d-qâ), and also improves in MRR and recall rates. This is expected since the ground-truth rather than predicted answers are included in the dialogue history (along with the ground-truth questions). The metrics sim cap,q and sim , on the other hand, show very little performance difference across the two evaluation settings. We also note that ranking performance is worse when both image i and caption c are excluded from condition variable. This does not, however, correlate with the CE and KL terms of the loss which are lower for a condition-less setting. We attribute this to the model being transformed from a CVAE to a VAE, hence lifting the burden of capturing the conditional posterior distribution (i.e. the KL is now between an unconditional q φ (z | x) and N (0, 1)). Interestingly, however, excluding either the image or the caption achieves similar performance to when both are included, indicating that the caption acts as a good textual proxy of the image (a reassurance of our sim c,q metric).

C. Extended Quantitative Results on 2VD task
Extending Tab. 4 in the main paper, Tab. 6 here shows results for B/B AR trained with permutations of the image i and caption c) (denoted by + if included in the condition, andotherwise). We note the decrease in CE and KL as conditions (i, c) are excluded from the model. This is expected since the task of dialogue generation is made simpler without the constrains of an explicit visual/textual condition.

D. Network architectures and training
The following section provides detailed descriptions of the architectures of our models A, B and B AR . The descriptions are dense but thorough. We also include further details of our training procedure. Where not explicitly noted, each convolutional layer is proceeded by a batch normalisation layer (with momentum = 0.001 and learnable parameters) and a ReLU activation.

Prior network
The prior neural network, parametrised by θ, takes as input the image i, the caption c and the dialogue context. For the model A, this context is h + t , containing the dialogue history up to t-1 and the current question q t . For models B/B AR , the dialogue context is the null set (h = ∅). To obtain the image representation, we scale and centre-cropped each image to 3 × 224 × 224 and feed it through VGG-16 [23]. The output of the penultimate layer is extracted and 2 -normalised (as in [6]) to obtain a 4096dimensional image feature vector. For the caption, we pass c through a pre-trained word2vec [18] model (we do not learn these word embeddings) to obtainc ∈ R 300×L where L is the maximum sentence length (L = 64). For the dialogue context (relevant only in the case of A) we pass the one-hot encoding of each word through a learnable word embedding module. We stack these embeddings as described in §3.1 of the main paper to obtainh + t ∈ R E×L×K , where E is the word embedding dimension (E = 256), L is the maximum sentence length (L = 64) and K is the number of dialogue entries at time t. We encode these inputs convolutionally to obtain y (the encoded condition) as follows:c is passed through a convolutional block (output size 64 × 8 × 8) and concatenated with the image feature vector (reshaped to 64×8×8). The concatenated output is passed through a convolutional block to obtain the jointly encoded image-caption (output size 64 × 8 × 8). If h = ∅, then the context is passed through a convolutional block (output size 64 × 8 × 8) and is concatenated with the encoded image-caption and passed through yet another convolutional block to get the encoded image-caption-context (output size 64 × 8 × 8). We call this the encoded condition y. The encoded condition y is then passed through a further convolutional block (output size 256 × 4 × 4) followed by two final convolutional layers (in parallel) to obtain µ p and log σ 2 p , respectively, the parameters of the conditional prior p θ (z | y). At this stage, µ p and Table 5: 1VD evaluation of B/B AR on VisDial (v0.9) test set. Results show ranking of answer candidates based on the S w2v scoring function. Note that d-qa indicates the iterative evaluation method when the ground-truth dialogue history is provided, while d-qâ, the iterative evaluation method when the ground-truth question and predicted answer history is provided (see §4.2). The + andindicate models trained with and without respective conditions, image i and caption c. log σ 2 p are both of size 512×1×1 (the latent dimensionality). At test time, a sample is obtained via z ∼ N (z; µ p , σ 2 p ) and is passed to the decoder in order to generate a sampleâ t (for A) ord (for B/B AR ).

Encoder network
The encoder network, parametrised by φ, takes x as input along with the encoded condition, y, obtained from the prior network. For model A, x = a t and y = {i, c, h + t }. For models B/B AR , x = d = (q t , a t ) T t=1 and y = {i, c}. In all models, x is passed through a learnable word embedding module, and the word embeddings stacked (see §3.1 in the main paper) to obtain x ∈ R E×L×M , where E = 256, L = 64 and M is the number of entries in x (for A, M = 1 and for B/B AR M = 2T ). In this way, we transform x into a single-channel answer 'image' in the case of A, and a multi-channel image of alternating questions and answers in the case of B/B AR .
x is then passed through a convolutional block (output size 64 × 8 × 8), the output of which is concatenated with y and forwarded through another convolutional block (output size 256 × 4 × 4). This output is forwarded through two final convolutional layers (in parallel) to obtain µ q and log σ 2 q , the parameters of the conditional latent posterior q φ (z | x, y).
Here µ q and log σ 2 q are both of size 512 × 1 × 1. At train time, the KL divergence term of the ELBO is computed using {µ q , σ q } (from the encoder network) and {µ p , σ p } (from the prior network).

Decoder network
The decoder network (for simplicity, the parameters of the prior and decoder network are subsumed into θ) takes as input a latent z and the encoded condition y. During training, z is sampled from a Gaussian parametrised by the µ q and exponentiated log σ 2 q outputs of the encoder network. This distribution is q φ (z | x, y). At test time, z is sampled from a Gaussian parametrised by the µ p and exponentiated log σ 2 p outputs of the prior network. This distribution is p θ (z | y). At both train and test time, we employ the commonly-used 're-parametrisation trick' [17] to compute the latent sample as z = µ + σ where ∼ N (0, 1) and µ and σ correspond to those derived from the encoder or prior network as described above.
The sample z is then transformed through a transposeconvolutional block (output size 64 × 8 × 8), concatenated with y and forwarded through a convolutional block (output size 64 × 8 × 8). This output is forwarded through a second transpose-convolutional block, producing an intermediate output volume of dimension M × E × L which we permute to match the size ofx. As before, E = 256, L = 64 and M = 1 (for A) or M = 2T (for B/B AR ).
Following this, our models diverge in architecture: A and B employ a standard linear layer which projects the E dimension of the intermediate output to the vocabulary size V . The B AR model instead employs an autoregressive module (detailed below) followed by this standard linear layer. At train time, the V -dimensional network output is softmaxed and used in the computation of the CE term of the ELBO. At test time, the argmax of the (softmax-ed) output is taken to be the index of the word token predicted. We share the weight matrices of the decoder's final linear Table 6: 2VD evaluation on VisDial (v0.9) test set for B/B AR models. Note that d left blank indicates the block evaluation method, when a whole dialogue is generated given only an image and its caption, while d-qâ indicates the iterative evaluation method when previously generated questions and answers are included in the dialogue history (see Section 4.2). The + andindicate models trained with and without respective conditions, image i and caption c. layer and the encoder and prior's learnable word embedding module (which are the same size by virtue of our network architecture) with the motivation that language encoders and decoders should share common word representations.

Autoregressive block
The autoregressive (AR) block (AR-N in Fig. 4 -bottom) in B AR 's decoder is inspired by PixelCNN [26] which sequentially predicts the pixels in an image along the two spatial dimensions. In the same fashion, we use an autoregressive approach to sequentially predict the next sentence (question or answer) in a dialogue. Since our framework is convolutional with sentences viewable as 'images', our approach can similarly be adapted from that of [26,9]. We first reshape the intermediate output of the decoder to E × L * M (essentially 'unravelling' the dialogue sequentially into a stack of its word embeddings). We then apply a size-preserving masked convolution to the reshaped output (followed by a learnable batch normalisation and a ReLU activation). We call this triplet an AR layer. The masked convolution of the AR layer ensures that future rows (i.e. future E-dimensional word embedding) are hidden in the prediction of the current row/word embedding. We apply N AR layers in this way with each layer taking in the output of the previous AR layer. Following the AR-N block, a linear layer projects the final output's E dimension to the vocabulary size V . We report numbers for N = {8, 10}. We base our implementation of the AR block on a publiclyavailable implementation of PixelCNN.

E. Dialogue preprocessing
The word vocabulary is constructed from the VisDial v0.9 [6] training dialogues (not including the candidate answers). The dialogues are preprocessed as follows: apostrophes are removed, numbers are converted to their worded equivalents, and all exchanges are made lower-case and either padded or truncated to a maximum sequence length (L = 64). The vocabulary is also filtered such that words with a frequency of <5 are removed and replaced with the UNK token. After pre-processing and filtering, the vocabulary size is V = 9710.

F. Extended Qualitative Results
We present additional qualitative results for the A model in Figs. 7 and 8 (1VD task) and for the B AR 10 model (under the block evaluation setting) in Figs. 9 and 10 (2VD task). Note that for both, different colours indicate generations (â t for A andd for B/B AR ) from different samples of z. In Figs. 9 and 10, whole generated dialogue blocks are shown with coloured sections indicating subsets exhibiting coherent question-answering and white sections indicating subsets that are not entirely coherent.


Universal Dependency Parsing from Scratch

This paper describes Stanford’s system at the CoNLL 2018 UD Shared Task. We introduce a complete neural pipeline system that takes raw text as input, and performs all tasks required by the shared task, ranging from tokenization and sentence segmentation, to POS tagging and dependency parsing. Our single system submission achieved very competitive performance on big treebanks. Moreover, after fixing an unfortunate bug, our corrected system would have placed the 2nd, 1st, and 3rd on the official evaluation metrics LAS, MLAS, and BLEX, and would have outperformed all submission systems on low-resource treebank categories on all metrics by a large margin. We further show the effectiveness of different model components through extensive ablation studies.

Introduction
Dependency parsing is an important component in various natural langauge processing (NLP) systems for semantic role labeling (Marcheggiani and Titov, 2017), relation extraction (Zhang et al., 2018), and machine translation (Chen et al., 2017). However, most research has treated dependency parsing in isolation, and largely ignored upstream NLP components that prepare relevant data for the parser, e.g., tokenizers and lemmatizers (Zeman et al., 2017). In reality, however, these upstream systems are still far from perfect.
To this end, in our submission to the CoNLL 2018 UD Shared Task, we built a raw-textto-CoNLL-U pipeline system that performs all tasks required by the Shared Task (Zeman et al., * These authors contributed roughly equally. 2018). 1 Harnessing the power of neural systems, this pipeline achieves competitive performance in each of the inter-linked stages: tokenization, sentence and word segmentation, partof-speech (POS)/morphological features (UFeats) tagging, lemmatization, and finally, dependency parsing. Our main contributions include: • New methods for combining symbolic statistical knowledge with flexible, powerful neural systems to improve robustness; • A biaffine classifier for joint POS/UFeats prediction that improves prediction consistency; • A lemmatizer enhanced with an edit classifier that improves the robustness of a sequenceto-sequence model on rare sequences; and • Extensions to our parser from (Dozat et al., 2017) to model linearization. Our system achieves competitive performance on big treebanks. After fixing an unfortunate bug, the corrected system would have placed the 2 nd , 1 st , and 3 rd on the official evaluation metrics LAS, MLAS, and BLEX, and would have outperformed all submission systems on low-resource treebank categories on all metrics by a large margin. We perform extensive ablation studies to demonstrate the effectiveness of our novel methods, and highlight future directions to improve the system. 2

System Description
In this section, we present detailed descriptions for each component of our neural pipeline system, namely the tokenizer, the POS/UFeats tagger, the lemmatizer, and finally the dependency parser.

Tokenizer
To prepare sentences in the form of a list of words for downstream processing, the tokenizer component reads raw text and outputs sentences in the CoNLL-U format. This is achieved with two subsystems: one for joint tokenization and sentence segmentation, and the other for splitting multiword tokens into syntactic words.
Tokenization and sentence segmentation. We treat joint tokenization and sentence segmentation as a unit-level sequence tagging problem. For most languages, a unit of text is a single character; however, in Vietnamese orthography, the most natural units of text are single syllables. 3 We assign one out of five tags to each of these units: end of token (EOT), end of sentence (EOS), multi-word token (MWT), multi-word end of sentence (MWS), and other (OTHER). We use bidirectional LSTMs (BiLSTMs) as the base model to make unit-level predictions. At each unit, the model predicts hierarchically: it first decides whether a given unit is at the end of a token with a score s (tok) , then classifies token endings into finer-grained categories with two independent binary classifiers: one for sentence ending s (sent) , and one for MWT s (MWT) .
Since sentence boundaries and MWTs usually require a larger context to determine (e.g., periods following abbreviations or the ambiguous word "des" in French), we incorporate token-level information into a two-layer BiLSTM as follows (see also Figure 1). The first layer BiLSTM operates directly on raw units, and makes an initial prediction over the categories. To help capture local unit patterns more easily, we also combine the firstlayer BiLSTM with 1-D convolutional networks, by using a one hidden layer convolutional network (CNN) with ReLU nolinearity at its first layer, giving an effect a little like a residual connection (He et al., 2016). The output of the CNN is simply added to the concatenated hidden states of the Bi-LSTM for downstream computation: where x is the input character representations, Step t Step t Figure 1: Illustration of the tokenizer/sentence segmenter model. Components in blue represent the gating mechanism between the two layers.
and W 1 contains the weights and biases for a linear classifier. 4 For each unit, we concatenate its trainable embedding with a four-dimensional binary feature vector as input, each dimension corresponding to one of the following feature functions: (1) does the unit start with whitespace; (2) does it start with a capitalized letter; (3) is the unit fully capitalized; and (4) is it purely numerical.
To incorporate token-level information at the second layer, we use a gating mechanism to suppress representations at non-token boundaries before propagating hidden states upward: where is an element-wise product broadcast over all dimensions of h 1 for each unit. This can be viewed as a simpler alternative to multiresolution RNNs (Serban et al., 2017), where the first-layer BiLSTM operates at the unit level, and the second layer operates at the token level. Unlike multi-resolution RNNs, this formulation is end-toend differentiable, and can more easily leverage efficient off-the-shelf RNN implementations.
To combine predictions from both layers of the BiLSTM, we simply sum the scores to obtain The final probability over the tags is then where p ±±± = σ(±s (tok) )σ(±s (sent) )σ(±s (MWT) ), and σ(·) is the logistic sigmoid function. p OTHER is simply σ(−s (tok) ). The model is trained to minimize the standard cross entropy loss.
Multi-word Token Expansion. The tokenizer/ sentence segmenter produces a collection of sentences, each being a list of tokens, some of which are labeled as multi-word tokens (MWTs). We must expand these MWTs into the underlying syntactic words they correspond to (e.g., "im" to "in dem" in German), in order for downstream systems to process them properly. To achieve this, we take a hybrid approach to combine symbolic statistical knowledge with the power of neural systems. The symbolic statistical side is a frequency lexicon. Many languages, like German, have only a handful of rules for expanding a few MWTs. We leverage this information by simply counting the number of times a MWT is expanded into different sequences of words in the training set, and retaining the most frequent expansion in a dictionary to use at test time. When building this dictionary, we lowercase all words in the expansions to improve robustness. However, this approach would fail for languages with rich clitics, a large set of unique MWTs, and/or complex rules for MWT expansion, such as Arabic and Hebrew. We capture this by introducing a powerful neural system.
Specifically, we train a sequence-to-sequence model using a BiLSTM encoder with an attention mechanism (Bahdanau et al., 2015) in the form of a multi-layer perceptron (MLP). Formally, the input multi-word token is represented by a sequence of characters x 1 , . . . , x I , and the output syntactic words are represented similarly as a sequence of characters y 1 , . . . , y J , where the words are separated by space characters. Inputs to the RNNs are encoded by a shared matrix of character embeddings E. Once the encoder hidden states h enc are obtained with a single-layer BiLSTM, each decoder step is unrolled as follows: (13) Here, w is a character index in the output vocabulary, y 0 a special start-of-sequence symbol in the vocabulary, and h dec 0 the concatenation of the last hidden states of each direction of the encoder.
To bring the symbolic and neural systems together, we train them separately and use the following protocol during evaluation: for each MWT, we first look it up in the dictionary, and return the expansion recorded there if one can be found. If this fails, we retry by lowercasing the incoming token. If that fails again, we resort to the neural system to predict the final expansion. This allows us to not only account for languages with flexible MWTs patterns (Arabic and Hebrew), but also leverage the training set statistics to cover both languages with simpler MWT rules, and MWTs in the flexible languages seen in the training set without fail. This results in a high-performance, robust system for multi-word token expansion.

POS/UFeats Tagger
Our tagger follows closely that of (Dozat et al., 2017), with a few extensions. As in that work, the core of the tagger is a highway BiLSTM (Srivastava et al., 2015) with inputs coming from the concatenation of three sources: (1) a pretrained word embedding, from the word2vec embeddings provided with the task when available (Mikolov et al., 2013), and from fastText embeddings otherwise (Bojanowski et al., 2017); (2) a trainable frequent word embedding, for all words that occurred at least seven times in the training set; and (3) a character-level embedding, generated from a unidirectional LSTM over characters in each word. UPOS is predicted by first transforming each word's BiLSTM state with a fully-connected (FC) layer, then applying an affine classifier: To predict XPOS, we similarly start with transforming the BiLSTM states with an FC layer. In order to further ensure consistency between the different tagsets (e.g., to avoid a VERB UPOS with an NN XPOS), we use a biaffine classifier, conditioned on a word's XPOS state as well as an embedding for its gold (at training time) or predicted (at inference time) UPOS tag y 163 UFeats is predicted analogously with separate parameters for each individual UFeat tag. The tagger is also trained to minimize the cross entropy loss. Some languages have composite XPOS tags, yielding a very large XPOS tag space (e.g., Arabic and Czech). For these languages, the biaffine classifier requires a prohibitively large weight tensor U (x) . For languages that use XPOS tagsets with a fixed number of characters, we classify each character of the XPOS tag in the same way we classify each UFeat. For the rest, instead of taking the biaffine approach, we simply share the FC layer between all three affine classifiers, hoping that the learned features for one will be used by another.

Lemmatizer
For the lemmatizer, we take a very similar approach to that of the multi-word token expansion component introduced in Section 2.1 with two key distinctions customized to lemmatization.
First, we build two dictionaries from the training set, one from a (word, UPOS) pair to the lemma, and the other from the word itself to the lemma. During evaluation, the predicted UPOS is used. When the UPOS-augmented dictionary fails, we fall back to the word-only dictionary before resorting to the neural system. In looking up both dictionaries, the word is never lowercased, because case information is more relevant in lemmatization than in MWT expansion.
Second, we enhance the neural system with an edit classifier that shortcuts the prediction process to accommodate rare, long words, on which the decoder is more likely to flounder. The concatenated encoder final states are put through an FC layer with ReLU nonlinearity and fed into a 3way classifier, which predicts whether the lemma is (1) exactly identical to the word (e.g., URLs and emails), (2) the lowercased version of the word (e.g., capitalized rare words in English that are not proper nouns), or (3) in need of the sequence-tosequence model to make more complex edits to the character sequence. During training time, we assign the labels to each word-lemma pair greedily in the order of identical, lowercase, and sequence decoder, and train the classifier jointly with the sequence-to-sequence lemmatizer. At evaluation time, predictions are made sequentially, i.e., the classifier first determines whether any shortcut can be taken, before the sequence decoder model is used if needed.

Dependency Parser
The dependency parser also follows that of (Dozat et al., 2017) with a few augmentations. The highway BiLSTM takes as input pretrained word embeddings, frequent word and lemma embeddings, character-level word embeddings, summed XPOS and UPOS embeddings, and summed UFeats embeddings. In (Dozat et al., 2017), unlabeled attachments are predicted by scoring each word i and its potential heads with a biaffine transformation where This approach, however, does not explicitly take into consideration relative locations of heads and dependents during prediction; instead, such predictive location information must be implicitly learned by the BiLSTM. Ideally, we would like the model to explicitly condition on (i − j), namely the dependent i and its potential head j's location relative to each other, in modeling p(y ij ). 5 Here, we motivate one way to build this into the model. First we factorize the relative location of word i and head j into their linear order and the distance between them, i.e., P (y ij |sgn(i − j), abs(i − j)), where sgn(·) is the sign function. Applying Bayes' rule and assuming conditional independence, we arrive at the following In a language where heads always follow their dependents, P (sgn(i − j) = 1|y ij ) would be extremely low, heavily penalizing rightward attachments. Similarly, in a language where dependencies are always short, P (abs(i−j) 0|y ij ) would be extremely low, penalizing longer edges.
P (y ij ) can remain the same as computed in Eq. (24). P (sgn(i − j)|y ij ) can be computed similarly with a deep biaffine scorer (cf. Eqs. (20)-(23)) over the recurrent states. This results in the score of j preceding i; flipping the sign wherever i precedes j turns this into the log odds of the ob-served linearization. Applying the sigmoid function then turns it into a probability: This can be effortlessly incorporated into the edge score by adding in the log of this probability − log(1 + exp(−s (l) ij )). Error is not backpropagated to this submodule through the final attachment loss; instead, it is trained with its own cross entropy, with error only computed on gold edges. This ensures that the model learns the conditional probability given a true edge, rather than just learning to predict the linear order of two words.
For P (abs(i − j)|y ij ), we use another deep biaffine scorer to generate a distance score. Distances are always no less than 1, so we apply 1 + softplus to predict the distance between i and j when there's an edge between them: where softplus(x) = log(1 + exp(x)). The distribution of edge lengths in the treebanks roughly follows a Zipfian distribution, to which the Cauchy distribution is closely related, only the latter is more stable for values at or near zero. Thus, rather than modeling the probability of an arc's length, we can use the Cauchy distribution to model the probability of an arc's error in predicted length, namely how likely it is for the predicted distance and the true distance to have a difference of δ When the difference δ (d) ij is small or zero, there will be effectively no penalty; but when the model expects a significantly longer or shorter arc than the observed distance between i and j, it is discouraged from assigning an edge between them. As with the linear order probability, the log of the distance probability is added to the edge score, and trained with its own cross-entropy on gold edges. 6 6 Note that the penalty assigned to the edge score in this way is proportional to ln δ ij ; using a Gamma At inference time, the Chu-Liu/Edmonds algorithm (Chu and Liu, 1965;Edmonds, 1967) is used to ensure a maximum spanning tree. Dependency relations are assigned to gold (at training time) or predicted (at inference time) edges y (e) i * using another deep biaffine classifier, following (Dozat et al., 2017) with no augmentations:

Training Details
Except where otherwise stated, our system is a pipeline: given a document of raw text, the tokenizer/sentence segmenter/MWT expander first splits it into sentences of syntactic words; the tagger then assigns UPOS, XPOS and UFeat tags to each word; the lemmatizer takes the predicted word and UPOS tag and outputs a lemma; finally, the parser takes all annotations as input and predicts the head and dependency label for each word. All components are trained with early stopping on the dev set when applicable. When a dev set is unavailable, we split the training set into an approximately 7-to-1 split for training and development. All components (except the dependency parser) are trained and evaluated on the development set assuming all related components had oracle implementations. This means the tokenizer/sentence segmenter assumes all correctly predicted MWTs will be correctly expanded, the MWT expander assumes gold word segmentation, and all downstream tasks assume gold word segmentation, along with gold annotations of all prerequisite tasks. The dependency parser is trained with predicted tags and morphological features from the POS/UFeats tagger.
Treebanks without training data. For treebanks without training data, we adopt a heuristic approach for finding replacements. Where a larger treebank in the same language is available (i.e., all PUD treebanks and Japanese-Modern), we used the models from the largest treebank available in that language. Where treebanks in related languages are available (as determined by language families from Wikipedia), we use models from the largest treebank in that related language. We or Poisson distribution to model the distance directly, or using a normal distribution instead of Cauchy, respectively, assigns penalties roughly proportional to δ ij . Thus, the Cauchy is more numerically stable during training. ended up choosing the models from English-EWT for Naija (an English-based pidgin), Irish-IDT for Breton (both are Celtic), and Norwegian-Nynorsk for Faroese (both are West Scandinavian). For Thai, since it uses a different script from all other languages, we use UDPipe 1.2 for all components.
Hyperparameters. The tokenizer/sentence segmenter uses BiLSTMs with 64d hidden states in each direction and takes 32d character embeddings as input. During training, we employ dropout to the input embeddings and hidden states at each layer with p = .33. We also randomly replace the input unit with a special <UNK> unit with p = .33, which would be used in place of any unseen input at test time. We add noise to the gating mechanism in Eq. (6) by randomly setting the gates to 1 with p = .02 and setting its temperature to 2 to make the model more robust to tokenization errors at test time. Optimization is performed with Adam (Kingma and Ba, 2015) with an initial learning rate of .002 for up to 20,000 steps, and whenever dev performance deteriorates, as is evaluated every 200 steps after the 2,000 th step, the learning rate is multiplied by .999. For the convolutional component we use filter sizes of 1 and 9, and for each filter size we use 64 channels (same as one direction in the BiLSTM). The convolutional outputs are concatenated in the hidden layer, before an affine transform is applied to serve as a residual connection for the BiLSTM. For the MWT expander, we use BiLSTMs with 256d hidden states in each direction as the encoder, a 512d LSTM decoder, 64d character embeddings as input, and dropout rate p = .5 for the inputs and hidden states. Models are trained up to 100 epochs with the standard Adam hyperparameters, and the learning rate is annealed similarly every epoch after the 15 th epoch by a factor of 0.9. Beam search of beam size 8 is employed in evaluation.
The lemmatizer uses BiLSTMs with 100d hidden states in each direction of the encoder, 50d character embeddings as input, and dropout rate p = .5 for the inputs and hidden states. The decoder is an LSTM with 200d hidden states. During training we jointly minimize (with equal weights) the cross-entropy loss of the edit classifier and the negative log-likelihood loss of the seq2seq lemmatizer. Models are trained up to 60 epochs with standard Adam hyperparameters.
The tagger and parser share most of their hyperparameters. We use 75d uncased frequent word and lemma embeddings, and 50d POS tag and UFeat embeddings. Pretrained embeddings and character-based word representations are both transformed to be 125d. During training, all embeddings are randomly replaced with a <drop> symbol with p = .33. We use 2-layer 200d BiL-STMs for the tagger and 3-layer 400d BiLSTMs for the parser. We employ dropout in all feedforward connections with p = .5 and all recurrent connections (Gal and Ghahramani, 2016) with p = .25 (except p = .5 in the tagger BiLSTM). All classifiers use 400d FC layers (except 100d for UFeats) with the ReLU nonlinearity. We train the systems with Adam (α = .003, β 1 = .9, β 2 = .95) until dev accuracy decreases, at which point we switch to AMSGrad (Reddi et al., 2018) until 3,000 steps pass with no dev accuracy increases.

Results
The main results are shown in Table 1. As can be seen from the table, our system achieves competitive performance on nearly all of the metrics when macro-averaged over all treebanks. Moreover, it achieves the top performance on several metrics when evaluated only on big treebanks, showing that our systems can effectively leverage statistical patterns in the data. Where it is not the top performing system, our system also achieved competitive results on each of the metrics on these treebanks. This is encouraging considering that our system is comprised of single-system components, whereas some of the best performing teams used ensembles (e.g., HIT-SCIR (Che et al., 2018)).
When taking a closer look, we find that our UFeats classifier is very accurate on these treebanks as well. Not only did it achieve the top performance on UFeats F 1 , but also it helped the parser achieve top MLAS as well on big treebanks, even when the parser is not the best-performing as evaluated by other metrics. We also note the contribution from our consistency modeling in the POS tagger/UFeats classifier: in both settings the individual metrics (UPOS, XPOS, and UFeats) achieve a lower advantage margin over the reference systems when compared to the AllTags metric, showing that these reference systems, though sometimes more accurate on each individual task, are not as consistent as our system overall.
The biggest disparity between the all-treebanks and big-treebanks results comes from sentence  segmentation. After inspecting the results on smaller treebanks and double-checking our implementation, we noticed issues with how we processed data in the tokenizer that negatively impacted generalization on these treebanks. 7 This is devastating for these treebanks, as all downstream components process words at the sentence level. We fixed this issue, and trained new tokenizers with all hyperparameters identical to our system at submission. We further built an unofficial evaluation pipeline, which we verified achieves the same evaluation results as the official system, and eval-7 Specifically, our tokenizer was originally designed to be aware of newlines (\n) in double newline-separated paragraphs, but we accidentally prepared training and dev sets for low resource treebanks by putting each sentence on its own line in the text file. This resulted in the sentence segmenter overfitting to relying on newlines. In later experiments, we replaced all in-paragraph whitespaces with space characters. uated our entire pipeline by only replacing the tokenizer. As is shown in Table 1, the resulting system (Stanford+) is much more accurate overall, and we would have ranked 2 nd , 1 st , and 3 rd on the official evaluation metrics LAS, MLAS, and BLEX, respectively. 8 On big treebanks, all metrics changed within only 0.02% F 1 and are thus not included. On small treebanks, however, this effect is more pronounced: as is shown in Table 2, our corrected system outperforms all submission systems on all official evaluation metrics on all low-resource treebanks by a large margin.

Analysis
In this section, we perform ablation studies on the new approaches we proposed for each component, and the contribution of each component to the final pipeline. For each component, we assume access to an oracle for all other components in the analysis, and show their efficacy on the dev sets. 9 For the ablations on the pipeline, we report macroaveraged F 1 on the test set.   Tokenizer. We perform ablation studies on the less standard components in the tokenizer, namely the gating mechanism in Eq. (6) (gating), the convolutional residual connections (conv), and the seq2seq model in the MWT expander (seq2seq), on all 61 big treebanks. As can be seen in Table 3, all but the gating mechanism make noticeable differences in macro F 1 . When taking a closer look, we find that both gating and conv show a mixed contribution to each treebank, and we could have improved overall performance further through treebank-level component selection. One surprising discovery is that conv greatly helps identify MWTs in Hebrew (+34.89 Words F 1 ) and sentence breaks in Ancient Greek-PROIEL (+18.77 Sents F 1 ). In the case of seq2seq, although the overall macro difference is small, it helps with the word segmentation performance on all treebanks where it makes any meaningful difference, most notably +10.08 on Hebrew and +4.19 on Arabic in Words F 1 (see also Figure 2). Finally, we note that dropout plays an important role in safeguarding the tokenizer from overfitting.  large, composite XPOS tagsets would incur prohibitive memory requirements. We therefore exclude treebanks that either have more than 250 XPOS tags or don't use them, leaving 36 treebanks for this analysis. We also measure consistency between tags by their pointwise mutual information where p c (X) is the accuracy of X. This quantifies (in nats) how much more likely it is to get all tags right than we would expect given their individual accuracies, if they were independent. As can be seen in Table 4, the added parameters do not affect UPOS performance significantly, but do help improve XPOS and UFeats prediction. Moreover, the biaffine classifier is markedly more consistent than the affine one with shared representations.
Lemmatizer. We perform ablation studies on three individual components in our lemmatizer: the edit classifier (edit), the sequence-to-sequence module (seq2seq) and the dictionaries (dictionaries). As shown in Table 5, we find that our lemmatizer with all components achieves the best overall performance. Specifically, adding the neural components (i.e., edit & seq2seq) drastically improves overall lemmatization performance over a simple dictionary-based approach (+6.77 F 1 ), and the gains are consistent over different treebank groups. While adding the edit classifier slightly decreases the F 1 score on small treebanks, it improves the performance on lowresource languages substantially (+0.91 F 1 ), and therefore leads to an overall gain of 0.11 F 1 . Tree-A n c i e n t G r e e k -P R O I E L

A r a b i c -P A D T
A n c i e n t G r e e k -P e r s e u s K o r e a n -K a i s t L a t i n -P R O I E L F i n n i s h -T D T F i n n i s h -F T B C r o a t i a n -S E T E s t o n i a n -E D T S l o v e n i a n -S S J B a s q u e -B D T S l o v a k -S N K L a t v i a n -L V T B C z e c h -P D T R o m a n i a n -R R T P o l i s h -L F G R u s s i a n -S y n T a g R u s C z e c h -F i c T r e e N o r w e g i a n -N y n o r s k N o r w e g i a n -B o k m a a l B u l g a r i a n -B T B F r e n c h -S p o k e n I t a l i a n -I S D T G a l i c i a n -C T G G e r m a n -G S D C a t a l a n -A n C o r a S w e d i s h -L i n E S H i n d i -H D T B F r e n c h -G S D S p a n i s h -A n C o r a U r d u -U D T B I t a l i a n -P o S T W I T A E n g l i s h -L i n E S P e r s i a n -S e r a j i E n g l i s h -G U M E n g l i s h -E W T J a p a n e s e -G S D I n d o n e s i a n -G S D C h i n e s e -G S D 0 0.5 1 Ratio of edit types seq2seq identity lowercase Figure 3: Edit operation types as output by the edit classifier on the official dev set. Due to space limit only treebanks containing over 120k dev words are shown and sorted by the ratio of seq2seq operation.  banks where the largest gains are observed include Upper Sorbian-UFAL (+4.55 F 1 ), Kurmanji-MG (+2.27 F 1 ) and English-LinES (+2.16 F 1 ). Finally, combining the neural lemmatizer with dictionaries helps capture common lemmatization patterns seen during training, leading to substantial improvements on all treebank groups.
To further understand the behavior of the edit classifier, for each treebank we present the ratio of all predicted edit types on dev set words in Figure 3. We find that the behavior of the edit classifier aligns well with linguistic knowledge. For example, while Ancient Greek, Arabic and Korean require a lot of complex edits in lemmatization, the vast majority of operations in Chinese and Japanese are simple identity mappings.
Dependency Parser. The main innovation for the parsing module is terms that model locations of a dependent word relative to possible head words in the sentence. Here we examine the impact of these terms, namely linearization (Eq. (28)) and distance (Eq. (34)). For this analysis, we exclude six treebanks with very small dev sets. As can be seen in Table 6, both terms contribute significantly to the final parser performance, with the distance term contributing slightly more.
Pipeline Ablation. We analyze the contribution of each pipeline component by incrementally replacing them with gold annotations and observing performance change. As shown in Figure 4, most downstream systems benefit moderately from gold sentence and word segmentation, while the parser S ta n fo rd + + g o ld to k + g o ld ta g + g o ld le m m a + g o ld p a rs e largely only benefits from improved POS/UFeats tagger performance (aside from BLEX, which is directly related to lemmatization performance and benefits notably). Finally, we note that the parser still is far from perfect even given gold annotations from all upstream tasks, but our components in the pipeline are very effective at closing the gap between predicted and gold annotations.

Conclusion & Future Directions
In this paper, we presented Stanford's submission to the CoNLL 2018 UD Shared Task. Our submission consists of neural components for each stage of a pipeline from raw text to dependency parses. The final system was very competitive on big treebanks; after fixing our preprocessing bug, it would have outperformed all official systems on all metrics for low-resource treebank categories. One of the greatest opportunities for further gains is through the use of context-sensitive word embeddings, such as ELMo (Peters et al., 2018) and ULMfit (Howard and Ruder, 2018). Although this requires a large resource investment, HIT-SCIR (Che et al., 2018) has shown solid improvements from incorporating these embeddings.