A matchmaking ratio (MMR) of greater than 1000 (low matchmaking ratios may occur due to broken replays or weak players).
<TABLE>Each replay contains global features, spatial features, cumulative scores, win/loss outcome, and ground truth actions [1]}. The global feature vector consists of the following sub-vectors: 1) frame id, 2) user's resource collection, 3) alerts received by the player, 4) upgrades applied by the player, 5) techniques researched by the player, 6) units and buildings owned by the player, and 7) enemy buildings and units observed by the player. These features are normalized in the interval [0, 1]. The spatial feature vector consists of the following features: 1) screen features, 2) mini-map features, 3) map height and width, 4) unit type, 5) unit density, and 6) unit relative. Fig. REF  depicts some of the spatial features. The dimension of these features is \(\mathbb {R}^{13\times 64\times 64}\) . Similar to global features, these features are also normalized to [0, 1]. Additional features include: 1) game reward i.e., final result (0 for loss and 1 for win), 2) cumulative score (not normalized), and 3) ground truth actions. Table REF  provides details on the action options and unit types available for each race.
<TABLE>Method
Figure REF  shows our proposed neural architecture. The backbone of the model consists of two branches - one for extracting information from global features and one for creating a compact representation of spatial features. A multi-layer perceptron (MLP) encoder, consisting of 3 linear layers with rectified linear units (ReLU), is used to learn a representation of the global features with a dimension of \(\mathbb {R}^{256}\) . Similarly, a convolutional neural network (CNN) encoder with 3 Conv2D layers followed a max pooling layer and ReLU activation is used to learn a compact representation of the spatial features. These representations of game states are then fed to a transformer encoder together with the positional encoding that preserves the ordering information. Following the discussion of [2]} and [3]}, we adopt the fixed positional encodings since the transformer model is permutation invariant.
The architecture of the transformer encoder is very similar to the standard encoder proposed by [4]} where each layer consists of a layer normalization module, a multi-headed self-attention module, and a feed forward network (FFN) module along with residual connections. The transformer encoder ingests a sequence of elements consisting of: 1) the output from the MLP encoder, 2) the output from CNN encoder (after being flattened to match the dimension of output of MLP encoder) and 3) positional encoding.
The transformer decoder also follows the standard implementation; however, we also include the positional encoding along with the sequence produced by the transformer encoder. The decoder receives the aggregated input consisting of 1) transformer encoder output, and 2) positional encoding. The softmax output from transformer decoder is fed to two separate prediction heads: Head Win and Head Build.
Our architecture contains six layers of each transformer encoder and transformer decoder with \(d_{model}\)  set to \(\mathbb {R}^{256}\) , number of self-attention heads to eight, a feed-forward dimensionality of \(\mathbb {R}^{1024}\) , dropout of 0.5 and ReLU activation.
Head Win is a 2-layer FFN with ReLU non-linearity, using a sigmoid output. It predicts the final result of the game (win or loss) given the current resources. This output is used to measure the accuracy of Global State Prediction (GSP). Like Head Win, Head Build is a 2-layer FFN with ReLU non-linearity and softmax output; it is used to evaluate the accuracy of Build Order Prediction (BOP).
Our full model also utilizes skip connections. The idea behind skip connections is to feed the output of a layer directly to a deeper layer while skipping a few layers in between. Skip connections provide some known advantages: 1) creation of an alternative gradient path, 2) prevention of the gradient vanishing problem, and 3) assistance with model convergence. Previous work has shown the effectiveness of skip connections in deep computer vision networks [5]}, [6]}. In our model, one skip connection is taken from the MLP encoder and is added to the output of the transformer before feeding to Head Win. The second skip connection is taken from the output of the CNN encoder and added to the output of the transformer to be fed to Head Build.
We compare our architecture to the highest performing baseline described by [1]} which uses a model that leverages both global and spatial features. Their model consists of couple of Conv2D layers to extract spatial features, a linear layer to extract global features; after combination, features are passed to a gated recurrent unit (GRU) with a hidden dimension of \(\mathbb {R}^{128}\) . A sigmoid output is used for global state prediction, and a softmax one for build order prediction.

Experiments
We implement our model in the PyTorch library. The model is trained with Adam optimizer [8]} by setting initial learning rate to \(1 \times 10^{-3}\) , \(\beta _1 = 0.9\)  and \(\beta _2 = 0.999\) . The learning rate is reduced by half every second epoch. The model was trained for 10 epochs with a batch size of 20 replays over 10 time steps on a Nvidia Tesla V100 GPU for each game.
For Head Win, we select the same loss function as was employed by the GRU baseline i.e., the Binary Cross Entropy:
\(\begin{split}\mathcal {L}_{global} = -\log (P(R = 1 | \Omega _t)) R_t \\ - \log (P(R = 0 | \Omega _t))(1 - R_t)\end{split}\) 
where \(\Omega _t\)  represents the observations at time \(t\)  and \(R\)  represents the final result of the game.
Unlike the baseline which employs Negative Log Likelihood Loss (NLL), Head Build employs the Cross Entropy Loss function. In this case, the loss function is given by the following equation:
\(\begin{aligned}\mathcal {L}_{build} = -\sum _{i=1}^{N} a_i \log (P(a_i | \Omega _t))\end{aligned}\) 
where \(a_i\)  is the ground truth action while \(P(a_i | \Omega _t)\)  represents the predicted action.
The final loss for our model is given by the combination across both tasks:
\(\begin{aligned}\mathcal {L} = \mathcal {L}_{global} + \mathcal {L}_{build}\end{aligned}\) 
In contrast, the baseline is done as two separate models with individual loss functions.

Results
This section presents a comprehensive evaluation of our proposed transformer vs. the best performing version of the baseline on two macromanagement prediction tasks. We also evaluate ablated versions of our model to show the relative contributions of the different components. Finally we illustrate the generalization power of the model in a transfer learning setting.
Both Global State Prediction (GSP) and Build Order Prediction (BOP) are challenging aspects of SC2 macromanagement. GSP looks at the game's current state and tries to predict the outcome of the game (win or loss) [9]}. BOP predicts the next production action (building, training, or researching) based on current state [10]}. Interestingly, all models perform better at BOP vs. GSP, despite the fact that a random chance guesser would perform better at predicting win/loss vs. predicting the next build action. We evaluate mean accuracy on both those tasks across all time steps using the provided test/train split in MSC.
Note that [1]} actually trained separate models for the GSP and BOP tasks; in contrast we train a single model with multi-headed output.
Our proposed transformer model captures the temporal relationships and context necessary to achieve better prediction power, and thus outperforms the baseline GRU model [1]} on both tasks. Table REF  provides a detailed breakdown of our results for every racial matchup.
<TABLE>Skip Connection
To evaluate the contributions of different aspects of our proposed neural architecture, we conduct an ablation study on the performance of different design choices. Our full model contains skip connections which bypass the transformer encoder and decoder stacks, providing the Head Win and Head Build with direct access to the original state vectors.
Table REF  shows the results of removing those skip connections. Skip connections do provide a slight advantage; our full model gained a couple of points over the one without skip connections on both tasks.
<TABLE>
Transformer Decoder
One research question that concerned us was whether the transformer decoder stack was redundant, since our proposed architecture includes output heads for computing win/loss and next build. Unlike many translation architectures that employ transformers, the loss function is not applied directly to the output of the decoder. Table REF  shows the results of omitting the decoder stack, which significantly reduces the number of model parameters. We observe that including the decoder stack results is small improvements in both prediction tasks.

Self-Attention Model
One drawback of the transformer is that it requires a significant computational investment to train the large number of parameters. We hypothesized that a simpler self-attention-based model may be able to capture contextual dependencies in SC2 production; for instance actions in build trees commonly co-occur with other actions. Similar to our transformer-based model, the self-attention model also consists of two branches: 1) an MLP encoder, and 2) a CNN encoder. We added a separate self-attention module for each branch. The output from both self-attention modules is then aggregated and is fed to 2-layer FFN which has two output heads: 1) one for global state and 2) one for build order prediction.
<TABLE>Table REF  shows that this version of the model performed surprisingly badly, underperforming both the transformer and the GRU baseline.
This indicates that in some ways the recency bias of the GRU is very useful since it focuses the model on a valuable temporal segment. In contrast the self-attention model fails to use temporal information effectively (disproving H2). The basic self-attention
module also lacks the positional encoding component of the transformer which helps the model learn ordering dependencies.

Transfer Learning
One benefit of the transformer model is its strong ability to generalize to new problems. We demonstrate this by training our proposed model in a transfer learning setting in which weights are learned from one racial matchup and transferred to another. To do this, we take a trained model from one pairing, freeze the MLP encoder, the CNN encoder, and the transformer, reinitialize both heads, train this model for 3 to 5 epochs and record the accuracy for both global state and build order. We keep the other training parameters fixed except the learning rate which is reduced to \(1 \times 10^{-5}\)  before starting the transfer learning experiments.
With transfer learning, our approach achieves even better performance in less time than the model trained from scratch on the same game. Table REF  presents the findings of our transfer learning experiments. In cases where both datasets share a common race, our model achieves even better results.

Conclusion and Future Work
This paper introduces a new neural architecture for macromanagement prediction tasks in SC2; our transformer-based architecture outperforms the GRU baselines on mean accuracy of global state and build order prediction on the MSC dataset. We demonstrate that the transformer is a superior option for capturing long-term temporal dependencies compared to recurrent neural networks which suffer from a recency bias (H1). However, the GRU is decisively superior to a simple self-attention system, disproving H2. Our ablative study justifies the inclusion of skip connections and transformer decoders. Our transformer is similar to the original one but includes a positional encoding input for the decoders. Our results on a transfer learning problem indicate that the model generalizes well (H3).
One of the strengths of the transformer is its ability to parallelize learning, unlike recurrent neural networks. Unfortunately the MSC dataset is not well designed for parallel loading, and we plan to tackle this problem in future work through the creation of a new dataset.
Another promising avenue for future research is coupling our transformer architecture with an RL macromanagement agent.

Acknowledgments
This research was partially supported by ARL STRONG W911NF-19-S-0001.
