The decoder of HEROSumm uses a standard decoder Transformer [1]}, as that in the baseline summarizer. There are studies of improving the decoder using the idea of MMR to avoid generating repeated phrases and increase the query-relevance and diversity of summaries. For example, [2]} propose a neural-based Seq2Seq framework which ensures that context vectors in attention mechanism are orthogonal to each other. Specifically, to alleviate the problem of repeating phrases in the summary, successive context vectors are treated as a sequence and a modified LSTM cell is used to compute the new state at each decoding time step. In decoding steps, the attention mechanism is used to focus on different portions of the query at different time steps.
