The Topological BERT: Transforming Attention into Topology for Natural Language Processing
In recent years, the introduction of the Transformer models sparked a
revolution in natural language processing (NLP). BERT was one of the first text
encoders using only the attention mechanism without any recurrent parts to
achieve state-of-the-art results on many NLP tasks.
  This paper introduces a text classifier using topological data analysis. We
use BERT's attention maps transformed into attention graphs as the only input
to that classifier. The model can solve tasks such as distinguishing spam from
ham messages, recognizing whether a sentence is grammatically correct, or
evaluating a movie review as negative or positive. It performs comparably to
the BERT baseline and outperforms it on some tasks.
  Additionally, we propose a new method to reduce the number of BERT's
attention heads considered by the topological classifier, which allows us to
prune the number of heads from 144 down to as few as ten with no reduction in
performance. Our work also shows that the topological model displays higher
robustness against adversarial attacks than the original BERT model, which is
maintained during the pruning process. To the best of our knowledge, this work
is the first to confront topological-based models with adversarial attacks in
the context of NLP.