Self-Attention and Transformer.
Transformer-based architectures, eschew the use of recurrence in neural networks and instead trust entirely on self-attention mechanisms to draw global dependencies between inputs and outputs.
Self-attention [1]} is particularly suitable for procedure planning, which can be seen as a sequence modeling task. Compared with Recurrent Neural Networks (RNNs), long short-term memory (LSTM) [2]} and gated recurrent neural networks [3]}, the advantages of self-attention includes avoiding compressing the whole past into a fixed-size hidden state, less total computational complexity per layer, and more parallelizable computations. In this paper, thanks to Transformers’ computational efficiency and scalability, we explore the possibility of marrying Transformer-based architecture for procedure planning.
