A gating operation merges the output of two encoder towers of a standard Transformer model for a single set of output predictions, as shown by Liu et al. [1]}. This operation combines and selects between multiple outputs from an encoder or decoder block. Gating also benefits from applying non-linear data processing when appropriate. Application of gating in various ways is a likely avenue for future innovations in Transformers, not only for time-series data but also for any other type of data. Several architectures use gating, including Gated Transformer Networks (GTN) [1]} and Temporal Fusion Transformers (TFT) [3]}. The proposed architecture of GTN uses gating techniques to incorporate information from two Transformer towers into the output. It does this by concatenating the output of each tower. TFT proposes Gated Linear Units (GLUs) [4]} to allow different parts of the network to be emphasized or suppressed based on the dataset.
