Here we extend the original Transformer [1]} and introduce the Tensorized Encoder Transformer (TENT) model suitable for learning complex underlying patterns of tensorial input data. The model processes inputs of 3D tensorial format \(X \in { \mathbb {R}^{T \times C \times F}}\)  where \(T\)  is the number of time steps, and \(C\)  and \(F\)  represent 2D features. 2D features of size \(C \times F\)  represent a matrix of weather variables such as temperature, humidity and wind speed measured in different geographical locations. For instance different cities along dimension \(C\)  and weather variables along dimension \(F\) . It should be noted that our model can also be used for applications other than weather forecasting, where the input data has a 3D tensorial format. It is an encoder-only Transformer architecture that consists of a positional encoding layer, three encoder layers followed by a fully-connected layer with linear activation as shown in Fig. REF . Each encoder layer uses the tensorial attention introduced in sections REF  and REF . Subsequent to the tensorial attention are a residual connection and a normalization layer. After the tensorial attention layer, each head has its own fully connected feed-forward network with two linear transformations and ReLU activations [2]}, followed by a second residual connection and normalization layer [1]}.
<FIGURE>