The learning rate is varied throughout the training process, first increasing linearly for the first training steps warmup steps = 4000 and then adjusted as described in <cite>(Vaswani et al., 2017)</cite> .