build order prediction: predicting which combat units will be researched, produced, and updated. Early prediction of the opponent's future army composition is a competitive advantage when making production decisions.

Vinyals et al.[1]} note that win-loss prediction in SC2 is challenging even for expert human players. The fog-of-war in SC2 that blocks players from seeing the map until they explore or build an area makes the current state only partially observable, rendering these prediction problems more difficult, as does the large state-action space [2]}, [3]}.
This paper introduces a transformer-based approach to global state and build order prediction that is inspired by the recent success of transformers in various natural language processing [4]} and computer vision tasks [5]}, [6]}. Unlike most sequence to sequence learning models, the transformer does not use a recurrent structure to capture time dependencies in sequences. Instead it uses an attention mechanism to decide which parts of the sequence are important [4]}. A positional encoding is added to the state embedding to preserve the ordering information.
Recurrent neural networks suffer from a recency bias, treating recent information as more important than actions in the distant past [8]}. Even gated units have difficulty retaining information over a long time horizon, since they have a probability of forgetting information at every time step. This property is useful for many tasks but poses a problem for macromanagement, which relies on the cumulative effects of decisions over a very long time horizon.
We hypothesize the following:

