<commit_before># Distributed Training

The `t2t-trainer` supports both synchronous and asynchronous distributed
training.

T2T uses TensorFlow Estimators and so distributed training is configured with
the `TF_CONFIG` environment variable that is read by the
[RunConfig](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/estimators/run_config.py)
along with a set of flags.

## `TF_CONFIG`

Both masters and parameter servers must have the `TF_CONFIG` environment
variable set.

The `TF_CONFIG` environment variable is a json-encoded string with the addresses
of the masters and parameter servers (in the `'cluster'` key) and the
identification of the current task (in the `'task'` key).

For example:

```
cluster = {
    'ps': ['host1:2222', 'host2:2222'],
    'master': ['host3:2222', 'host4:2222', 'host5:2222']
}
os.environ['TF_CONFIG'] = json.dumps({
    'cluster': cluster,
    'task': {'type': 'master', 'index': 1},
    'environment': 'cloud',
})
```

## Command-line flags<commit_msg>distributed: reference to tf.contrib.learn is deprecated. Link to tf.estimator.RunConfig is used instead. Introduced with https://github.com/tensorflow/tensorflow/commit/c7caa2d87daa37b66811ac99f997ad02acd4ecc8<commit_after># Distributed Training

The `t2t-trainer` supports both synchronous and asynchronous distributed
training.

T2T uses TensorFlow Estimators and so distributed training is configured with
the `TF_CONFIG` environment variable that is read by the
[RunConfig](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/estimator/run_config.py)
along with a set of flags.

## `TF_CONFIG`

Both masters and parameter servers must have the `TF_CONFIG` environment
variable set.

The `TF_CONFIG` environment variable is a json-encoded string with the addresses
of the masters and parameter servers (in the `'cluster'` key) and the
identification of the current task (in the `'task'` key).

For example:

```
cluster = {
    'ps': ['host1:2222', 'host2:2222'],
    'master': ['host3:2222', 'host4:2222', 'host5:2222']
}
os.environ['TF_CONFIG'] = json.dumps({
    'cluster': cluster,
    'task': {'type': 'master', 'index': 1},
    'environment': 'cloud',
})
```

## Command-line flags
{'fraction_non_alphanumeric': 0.12299465240641712, 'fraction_numerical': 0.0338680926916221, 'mean_word_length': 5.2534818941504176, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2733655', 'n_tokens_mistral': 736, 'n_tokens_neox': 677, 'n_words': 227}