Long-context Window SupportWe implement and improve computation-communication overlapping, sequence parallelism, and communication compression to support up to 200K context length continue pretraining and finetuning. Our method to scale the context length to 200K is _solely_ based on engineering, that is to say, we do not modify the model architecture like sparse, local, or sliding window attention - the model remains using the full attention even the input is 200K.