The Transformer model generally consists of encoder and decoder blocks followed by linear and softmax layers for decision making. One approach referred to as Simply Attend and Diagnose (SAnD) replaces the decoder block with a dense interpolation layer to incorporate temporal order into the model’s processing [1]}. This approach does not utilize output embedding, cutting down the number of total layers in the model. Without the decoder, the model needs a way to process the outputs of the encoder block to be input into the linear layers. Simple concatenation leads to poor prediction accuracy; therefore [1]} developed a dense interpolation algorithm with hyperparameters that can be tuned to improve performance.
