In natural language processing, sequence labeling tasks involve labeling every word in a sequence of text with a linguistic tag. These tasks were traditionally performed using shallow linear models such as hidden Markov models (HMMs) [1]}, [2]} and linear chain conditional random fields (CRFs) [3]}, [4]}, [5]}. These approaches model the dependencies between adjacent word-level labels. However when predicting the label for a given word, they do not directly incorporate information from the surrounding words in the sentence (known as `context'). As a result, linear chain CRFs combined with deeper models which do use such contextual information (e.g. convolutional and recurrent networks) gained popularity [6]}, [7]}, [8]}, [9]}.
