The authors in [1]} used tensorized transformer to compress model parameters and increase performance. In particular, [1]} introduced Multi-linear attention for language modeling approaches, which uses Tucker decomposition [3]}, [4]} in Single-block attention to reconstruct the scaled dot-product attention and Block-Term Tensor Decomposition [5]} for the multi-head attention mechanism. This paper proposes a Tensorized Encoder Transformer (TENT) for temperature forecasting task. The model is equipped with a new tensorial attention mechanism. As opposed to [1]}, our proposed TENT model receives 3-dimensional tensors as input and maintains the tensorial format in the calculation of the self-attention layer. Thanks to the introduced tensorial self-attention mechanism, TENT model is able to capture the dependencies between time steps and weather stations. In the subsequent sections, the proposed model will be explained in detail.
