Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles
Linking neural representations to linguistic factors is crucial in order to
build and analyze NLP models interpretable by humans. Among these factors,
syntactic roles (e.g. subjects, direct objects,$\dots$) and their realizations
are essential markers since they can be understood as a decomposition of
predicative structures and thus the meaning of sentences. Starting from a deep
probabilistic generative model with attention, we measure the interaction
between latent variables and realizations of syntactic roles and show that it
is possible to obtain, without supervision, representations of sentences where
different syntactic roles correspond to clearly identified different latent
variables. The probabilistic model we propose is an Attention-Driven
Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based
machine translation models, ADVAEs enable the analysis of the interactions
between latent variables and input tokens through attention. We also develop an
evaluation protocol to measure disentanglement with regard to the realizations
of syntactic roles. This protocol is based on attention maxima for the encoder
and on latent variable perturbations for the decoder. Our experiments on raw
English text from the SNLI dataset show that $\textit{i)}$ disentanglement of
syntactic roles can be induced without supervision, $\textit{ii)}$ ADVAE
separates syntactic roles better than classical sequence VAEs and Transformer
VAEs, $\textit{iii)}$ realizations of syntactic roles can be separately
modified in sentences by mere intervention on the associated latent variables.
Our work constitutes a first step towards unsupervised controllable content
generation. The code for our work is publicly available.