Generating token tags with a linear layer has one big flaw: predictions for each token are made independently. CRF mitigates this problem and improves the metrics, but we decided to employ LaserTagger - an autoregressive transformer decoder from [1]}. One of the critical ideas of LaserTagger is directly consuming corresponding encoder activation without learning encoder-decoder attention weights. LaserTagger achieved an even stronger baseline with an F-score of 42% on development set, which at the time was only several points away from top-performing teams. It was decided to keep this model and extensively tune it during further experiments.
<FIGURE>